Original: Learning Globally Smooth Functions on Manifolds
Revised: Learning Globally Smooth Functions on Manifolds

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Smoothness and low dimensional structures play central roles in improving generalization and stability in learning and statistics.
Revised: Smoothness and low dimensional structures play central roles in improving generalization and stability in learning and statistics.

Original: The combination of these properties has led to many advances in semi-supervised learning, generative modeling, and control of dynamical systems.
Revised: The combination of these properties has led to many advances in semi-supervised learning, generative modeling, and control of dynamical systems.

Original: However, learning smooth functions is generally challenging, except in simple cases such as learning linear or kernel models.
Revised: However, learning smooth functions is generally challenging, except in simple cases such as learning linear or kernel models.

Original: Typical methods are either too conservative, relying on crude upper bounds such as spectral normalization, too lax, penalizing smoothness on average, or too computationally intensive, requiring the solution of large-scale semi-definite programs.
Revised: Typical methods are either too conservative, relying on crude upper bounds such as spectral normalization, too lax, penalizing smoothness on average, or too computationally intensive, requiring the solution of large-scale semi-definite programs.

Original: These issues are only exacerbated when trying to simultaneously exploit low dimensionality using, e.g., manifolds.
Revised: These issues are only exacerbated when trying to simultaneously exploit low dimensionality using, e.g., manifolds.

Original: This work proposes to overcome these obstacles by combining techniques from semi-infinite constrained learning and manifold regularization.
Revised: This work proposes to overcome these obstacles by combining techniques from semi-infinite constrained learning and manifold regularization.

Original: To do so, it shows that, under typical conditions, the problem of learning a Lipschitz continuous function on a manifold is equivalent to a dynamically weighted manifold regularization problem.
Revised: To do so, it shows that, under typical conditions, the problem of learning a Lipschitz continuous function on a manifold is equivalent to a dynamically weighted manifold regularization problem.

Original: This observation leads to a practical algorithm based on a weighted Laplacian penalty whose weights are adapted using stochastic gradient techniques.
Revised: This observation leads to a practical algorithm based on a weighted Laplacian penalty whose weights are adapted using stochastic gradient techniques.

Original: We prove that, under mild conditions, this method estimates the Lipschitz constant of the solution, learning a globally smooth solution as a byproduct.
Revised: We prove that, under mild conditions, this method estimates the Lipschitz constant of the solution, learning a globally smooth solution as a byproduct.

Original: Numerical examples illustrate the advantages of using this method to impose global smoothness on manifolds as opposed to imposing smoothness on average.
Revised: Numerical examples illustrate the advantages of using this method to impose global smoothness on manifolds as opposed to imposing smoothness on average.

Original:  Introduction
Revised:  Introduction

Original: Learning smooth functions has been shown to be advantageous in general and is of particular interest in physical systems.
Revised: Learning smooth functions has been shown to be advantageous in general and is of particular interest in physical systems.

Original: This is because of the general observation that close input features tend to be associated with close outputs and of the particular fact that in physical systems Lipschitz continuity of input-output maps translates to stability and safety (Oberman and Calder, 2018; Finlay et al., 2018a; Couellan, 2021; Finlay et al., 2018b; Pauli et al., 2021;
Revised: This is because of the general observation that close input features tend to be associated with close outputs and of the particular fact that in physical systems Lipschitz continuity of input-output maps translates to stability and safety (Oberman and Calder, 2018; Finlay et al., 2018a; Couellan, 2021; Finlay et al., 2018b; Pauli et al., 2021;

Original: Krishnan et al., 2020; Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021).
Revised: Krishnan et al., 2020; Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021).

Original: To learn smooth functions one can require the parameterization to be smooth.
Revised: To learn smooth functions one can require the parameterization to be smooth.

Original: Such is the idea, e.g., of spectral normalization of weights in neural networks (Miyato et al., 2018;
Revised: Such is the idea, e.g., of spectral normalization of weights in neural networks (Miyato et al., 2018;

Original: Zhao and Liu, 2020).
Revised: Zhao and Liu, 2020).

Original: Smooth parameterizations have the advantage of being globally smooth, but they may be restrictive because they impose smoothness for inputs that are not necessarily realized in the data.
Revised: Smooth parameterizations have the advantage of being globally smooth, but they may be restrictive because they impose smoothness for inputs that are not necessarily realized in the data.

Original: This drawback motivates the use of Lipschitz penalties in risk minimization (Oberman and Calder, 2018; Finlay et al., 2018a; Couellan, 2021; Pauli et al., 2021; Bungert et al., 2021), which offers the opposite tradeoff.
Revised: This drawback motivates the use of Lipschitz penalties in risk minimization (Oberman and Calder, 2018; Finlay et al., 2018a; Couellan, 2021; Pauli et al., 2021; Bungert et al., 2021), which offers the opposite tradeoff.

Original: Since penalties encourage but do not enforce small Lipschitz constants, we may learn functions that are smooth on average, but with no global guarantees of smoothness at every point in the support of the data.
Revised: Since penalties encourage but do not enforce small Lipschitz constants, we may learn functions that are smooth on average, but with no global guarantees of smoothness at every point in the support of the data.

Original: Formulations that guarantee global smoothness can be obtained if the risk minimization problem is modified by the addition of a Lipschitz constant constraint (Krishnan et al., 2020;
Revised: Formulations that guarantee global smoothness can be obtained if the risk minimization problem is modified by the addition of a Lipschitz constant constraint (Krishnan et al., 2020;

Original: Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021).
Revised: Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021).

Original: This yields formulations that guarantee Lipschitz smoothness in all possible inputs without the drawback of enforcing smoothness outside of the input data distribution.
Revised: This yields formulations that guarantee Lipschitz smoothness in all possible inputs without the drawback of enforcing smoothness outside of the input data distribution.

Original: Several empirical studies (Krishnan et al., 2020; Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021) have demonstrated the advantage of imposing global smoothness constraints only on observed inputs.
Revised: Several empirical studies (Krishnan et al., 2020; Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021) have demonstrated the advantage of imposing global smoothness constraints only on observed inputs.

Original: [Table]
Revised: [Table]

Original: (d) Ambient Regularization
Revised: (d) Ambient Regularization

Original: In this paper we exploit the fact that data can be often modeled as points in a low-dimensional manifold.
Revised: In this paper we exploit the fact that data can be often modeled as points in a low-dimensional manifold.

Original: We therefore consider manifold Lipschitz constants in which function smoothness is assessed with respect to distances measured over the data manifold (Definition 1).
Revised: We therefore consider manifold Lipschitz constants in which function smoothness is assessed with respect to distances measured over the data manifold (Definition 1).

Original: Although this looks like a minor difference, controlling Lipschitz constants over data manifolds is quite different from controlling Lipschitz constants in the ambient space.
Revised: Although this looks like a minor difference, controlling Lipschitz constants over data manifolds is quite different from controlling Lipschitz constants in the ambient space.

Original: In Figure 1 we look at a classification problem with classes arranged in two separate half moons.
Revised: In Figure 1 we look at a classification problem with classes arranged in two separate half moons.

Original: Constraining
Revised: Constraining

Original: Lipschitz constants in the ambient space effectively assumes the underlying data is uniformly distributed in space
Revised: Lipschitz constants in the ambient space effectively assumes the underlying data is uniformly distributed in space

Original: [cf.
Revised: [cf.

Original: Figure 1-(d)].
Revised: Figure 1-(d)].

Original: Constraining Lipschitz constants in the data manifold, however, properly accounts for the data distribution [cf.
Revised: Constraining Lipschitz constants in the data manifold, however, properly accounts for the data distribution [cf.

Original: Figure 1-(a)].
Revised: Figure 1-(a)].

Original: This example also illustrates how constraining manifold Lipschitz constants is related to manifold regularization (Belkin et al., 2005; Niyogi, 2013;
Revised: This example also illustrates how constraining manifold Lipschitz constants is related to manifold regularization (Belkin et al., 2005; Niyogi, 2013;

Original: Li et al., 2022).
Revised: Li et al., 2022).

Original: The difference is that manifold regularization penalizes the average norm of the manifold gradient.
Revised: The difference is that manifold regularization penalizes the average norm of the manifold gradient.

Original: This distinction is significant because regularizing is more brittle than imposing constraints.
Revised: This distinction is significant because regularizing is more brittle than imposing constraints.

Original: In the example in Figure 1, manifold regularization fails to separate the dataset when the moons are close [cf.
Revised: In the example in Figure 1, manifold regularization fails to separate the dataset when the moons are close [cf.

Original: Figure 1-(c), bottom].
Revised: Figure 1-(c), bottom].

Original: Classification with a manifold Lipschitz constant constraint is more robust to this change in the data distribution [cf.
Revised: Classification with a manifold Lipschitz constant constraint is more robust to this change in the data distribution [cf.

Original: Figure 1-(a), bottom].
Revised: Figure 1-(a), bottom].

Original: Global constrains in the manifold gradient yield a statistical constrained learning problem with an infinite and dense number of constraints.
Revised: Global constrains in the manifold gradient yield a statistical constrained learning problem with an infinite and dense number of constraints.

Original: This is a challenging problem to approximate and solve.
Revised: This is a challenging problem to approximate and solve.

Original: Here, we approach the solution of this problem in the Lagrangian dual domain and establish connections with manifold regularization that allow for the use of point cloud
Revised: Here, we approach the solution of this problem in the Lagrangian dual domain and establish connections with manifold regularization that allow for the use of point cloud

Original: Laplacians.
Revised: Laplacians.

Original: Our specific contributions are the following:
Revised: Our specific contributions are the following:

Original: (C1)
Revised: (C1)

Original: We introduce a constrained statistical risk minimization problem in which we learn a function that: (i) attains a target loss and (ii) attains the smallest possible manifold Lipschitz constant among these functions that satisfy the target loss (Section 2).
Revised: We introduce a constrained statistical risk minimization problem in which we learn a function that: (i) attains a target loss and (ii) attains the smallest possible manifold Lipschitz constant among these functions that satisfy the target loss (Section 2).

Original: (C2)
Revised: (C2)

Original: We introduce the Lagrangian dual problem and show that its empirical version is a statistically consistent approximation of the primal.
Revised: We introduce the Lagrangian dual problem and show that its empirical version is a statistically consistent approximation of the primal.

Original: This results does not require the learning parametrization to be convex (Section 3.1).
Revised: This results does not require the learning parametrization to be convex (Section 3.1).

Original: (C3)
Revised: (C3)

Original: We generalize results from the manifold regularization literature to show that under regularity conditions, the evaluation of manifold Lipschitz constants can be recast in a more amenable form utilizing a weighted point cloud Laplacian (Proposition 3 in Section 3.2).
Revised: We generalize results from the manifold regularization literature to show that under regularity conditions, the evaluation of manifold Lipschitz constants can be recast in a more amenable form utilizing a weighted point cloud Laplacian (Proposition 3 in Section 3.2).

Original: (C4)
Revised: (C4)

Original: We present a dual ascent algorithm to find optimal multipliers.
Revised: We present a dual ascent algorithm to find optimal multipliers.

Original: The function that attains the target loss and minimizes the manifold Lipschitz constant follows as a byproduct (Section 3.3).
Revised: The function that attains the target loss and minimizes the manifold Lipschitz constant follows as a byproduct (Section 3.3).

Original: (C5)
Revised: (C5)

Original: We illustrate the merits of learning with global manifold smoothness guarantees with respect to ambient space and standard manifold regularization through two examples: (i) learning robot navigation policies in a space with obstacles and (ii) learning model mismatches in differential drive steering over non-ideal surfaces (Section 4)
Revised: We illustrate the merits of learning with global manifold smoothness guarantees with respect to ambient space and standard manifold regularization through two examples: (i) learning robot navigation policies in a space with obstacles and (ii) learning model mismatches in differential drive steering over non-ideal surfaces (Section 4)

Original: Related Work
Revised: Related Work

Original: This paper is at the intersection of learning with Lipschitz constant constraints (Oberman and Calder, 2018; Finlay et al., 2018a; Couellan, 2021; Pauli et al., 2021; Bungert et al., 2021;
Revised: This paper is at the intersection of learning with Lipschitz constant constraints (Oberman and Calder, 2018; Finlay et al., 2018a; Couellan, 2021; Pauli et al., 2021; Bungert et al., 2021;

Original: Miyato et al., 2018; Zhao and Liu, 2020; Krishnan et al., 2020; Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021) and manifold regularization (Belkin et al., 2005; Niyogi, 2013; Li et al., 2022;
Revised: Miyato et al., 2018; Zhao and Liu, 2020; Krishnan et al., 2020; Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021) and manifold regularization (Belkin et al., 2005; Niyogi, 2013; Li et al., 2022;

Original: Hein et al., 2005; Belkin and Niyogi, 2005).
Revised: Hein et al., 2005; Belkin and Niyogi, 2005).

Original: Relative to the literature on learning with Lipschitz constraints we offer the ability to leverage data manifolds.
Revised: Relative to the literature on learning with Lipschitz constraints we offer the ability to leverage data manifolds.

Original: Since data manifolds are often characterized with unlabeled data (Kejani et al., 2020; Belkin and Niyogi, 2004; Jiang et al., 2019; Kipf and Welling, 2016; Yang et al., 2016; Zhu, 2005; Lecouat et al., 2018; Ouali et al., 2020; Cabannes et al., 2021), similar to these works, we utilize the point cloud Laplacian technique to compute the integral of the norm of the gradient.
Revised: Since data manifolds are often characterized with unlabeled data (Kejani et al., 2020; Belkin and Niyogi, 2004; Jiang et al., 2019; Kipf and Welling, 2016; Yang et al., 2016; Zhu, 2005; Lecouat et al., 2018; Ouali et al., 2020; Cabannes et al., 2021), similar to these works, we utilize the point cloud Laplacian technique to compute the integral of the norm of the gradient.

Original: Relative to the literature on manifold regularization we offer global smoothness assurances instead of an average penalty of large manifold gradients.
Revised: Relative to the literature on manifold regularization we offer global smoothness assurances instead of an average penalty of large manifold gradients.

Original: In (Krishnan et al., 2020), the space is discretized maintaining convexity, and at the expense of deriving a sensitive solution, their approach leads to a large optimization problem.
Revised: In (Krishnan et al., 2020), the space is discretized maintaining convexity, and at the expense of deriving a sensitive solution, their approach leads to a large optimization problem.

Original: Similar to us, (Krishnan et al., 2020) poses the problem of minimizing a Lipschtiz constant, however, they utilize a softer surrogate (i.e. p -norm loss) which is a smoother version of the Lipschitz constant.
Revised: Similar to us, (Krishnan et al., 2020) poses the problem of minimizing a Lipschtiz constant, however, they utilize a softer surrogate (i.e. p -norm loss) which is a smoother version of the Lipschitz constant.

Original: Their approach therefore, tradeoffs numerical stability (small p ) with accurate Lipschitz constant estimation ( p = ∞ ).
Revised: Their approach therefore, tradeoffs numerical stability (small p ) with accurate Lipschitz constant estimation ( p = ∞ ).

Original: We do not work with surrogates, and we seek to minimize the maximum norm of the gradient, utilizing an epigraph technique.
Revised: We do not work with surrogates, and we seek to minimize the maximum norm of the gradient, utilizing an epigraph technique.

Original:  Global Constraining of Manifold Lipschitz Constants
Revised:  Global Constraining of Manifold Lipschitz Constants

Original: We consider data pairs ( x, y ) in which the input features x ∈ M ⊂ R N lie in a compact oriented Riemannian manifold M and the output features are real valued y ∈ R .
Revised: We consider data pairs ( x, y ) in which the input features x ∈ M ⊂ R D lie in a compact oriented Riemannian manifold M and the output features are real valued y ∈ R .

Original: We study the regression problem of finding a function f θ : M → R , parameterized by θ ∈ Θ ⊂ R Q that minimizes the expectation of a nonnegative loss ℓ : R × R → R + , where ℓ ( f θ ( x ) , y ) represents the loss of predicting output f θ ( x ) when the world realizes the pair ( x, y ) .
Revised: We study the regression problem of finding a function f θ : M → R , parameterized by θ ∈ Θ ⊂ R Q that minimizes the expectation of a nonnegative loss ℓ : R × R → R + , where ℓ ( f θ ( x ) , y ) represents the loss of predicting output f θ ( x ) when the world realizes the pair ( x, y ) .

Original: Data pairs ( x, y ) are drawn according to an unknown probability distribution p ( x, y ) on M × R which we can factor as p ( x, y )
Revised: Data pairs ( x, y ) are drawn according to an unknown probability distribution p ( x, y ) on M × R which we can factor as p ( x, y )

Original: = p ( x ) p ( y | x ) .
Revised: = p ( x ) p ( y | x ) .

Original: We are interested in learning smooth functions, i.e. functions with controlled variability over the manifold M .
Revised: We are interested in learning smooth functions, i.e. functions with controlled variability over the manifold M .

Original: We therefore let ∇ M f θ ( x ) represent the manifold gradient of f θ and introduce the following definition.
Revised: We therefore let ∇ M f θ ( x ) represent the manifold gradient of f θ and introduce the following definition.

Original: Definition 1 (Manifold Lipschitz Constant) .
Revised: Definition 1 (Manifold Lipschitz Constant) .

Original: Given a Riemannian manifold M , the function f θ : M → R is said to be L -Lipschitz continuous if there exists a strictly positive constant
Revised: Given a Riemannian manifold M , the function f θ : M → R is said to be L -Lipschitz continuous if there exists a strictly positive constant

Original: L > 0 such that for all pairs of points x 1 , x 2 ∈ M ,
Revised: L > 0 such that for all pairs of points x 1 , x 2 ∈ M ,

Original: [Equation]
Revised: [Equation]

Original: where d M ( x 1 , x 2 ) denotes the distance between x 1 and x 2 in the manifold M .
Revised: where d M ( x 1 , x 2 ) denotes the distance between x 1 and x 2 in the manifold M .

Original: If the function f θ is differentiable, (1) is equivalent to requiring the gradient norm to be bounded by L ,
Revised: If the function f θ is differentiable, (1) is equivalent to requiring the gradient norm to be bounded by L ,

Original: [Equation]
Revised: [Equation]

Original: With definition 1 in place and restricting attention to differentiable functions f θ our stated goal of learning functions f θ with controlled variability over the manifold M can then be written as
Revised: With definition 1 in place and restricting attention to differentiable functions f θ our stated goal of learning functions f θ with controlled variability over the manifold M can then be written as

Original: [Equation]
Revised: [Equation]

Original: In this formulation the statistical loss E p ( x,y )
Revised: In this formulation the statistical loss E p ( x,y )

Original: [ ℓ ( f θ ( x ) , y )] is required to be below a target level ϵ .
Revised: [ ℓ ( f θ ( x ) , y )] is required to be below a target level ϵ .

Original: Of all functions f θ that can satisfy this loss requirement, Problem (3) defines as optimal the one whose Lipschitz constant L = √ ρ is the smallest.
Revised: Of all functions f θ that can satisfy this loss requirement, Problem (3) defines as optimal the one whose Lipschitz constant L = √ ρ is the smallest.

Original: The goal of this paper is to develop methodologies to solve (3) when the data distribution and the manifold are unknown.
Revised: The goal of this paper is to develop methodologies to solve (3) when the data distribution and the manifold are unknown.

Original: To characterize the distribution we are given sample pairs ( x i , y i ) drawn from the joint distribution p ( x, y ) .
Revised: To characterize the distribution we are given sample pairs ( x i , y i ) drawn from the joint distribution p ( x, y ) .

Original: To characterize the manifold we are given samples z i drawn from the marginal distribution p ( x ) .
Revised: To characterize the manifold we are given samples z i drawn from the marginal distribution p ( x ) .

Original: This includes the samples x i from the (labeled) data pairs ( x i , y i ) and may also include (unlabeled) samples z i .
Revised: This includes the samples x i from the (labeled) data pairs ( x i , y i ) and may also include (unlabeled) samples z i .

Original: It is interesting to observe that (3) shows that the problems of manifold regularization (Belkin et al., 2005; Niyogi, 2013; Kejani et al., 2020; Li et al., 2022) and Lipschitz constant control (Oberman and Calder, 2018; Finlay et al., 2018a; Couellan, 2021; Finlay et al., 2018b;
Revised: It is interesting to observe that (3) shows that the problems of manifold regularization (Belkin et al., 2005; Niyogi, 2013; Kejani et al., 2020; Li et al., 2022) and Lipschitz constant control (Oberman and Calder, 2018; Finlay et al., 2018a; Couellan, 2021; Finlay et al., 2018b;

Original: Pauli et al., 2021; Krishnan et al., 2020; Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021) are related.
Revised: Pauli et al., 2021; Krishnan et al., 2020; Shi et al., 2019; Lindemann et al., 2021; Arghal et al., 2021) are related.

Original: This connection is important to understand the merit of (3).
Revised: This connection is important to understand the merit of (3).

Original: To explain this better observe that there are three motivations for the problem formulation in (3): (i)
Revised: To explain this better observe that there are three motivations for the problem formulation in (3): (i)

Original: It is often the case that if samples x 1 and x 2 are close, then the conditional distributions p ( y | x 1 ) and p ( y | x 2 ) are close as well.
Revised: It is often the case that if samples x 1 and x 2 are close, then the conditional distributions p ( y | x 1 ) and p ( y | x 2 ) are close as well.

Original: A function f θ with small Lipschitz constant leverages this property.
Revised: A function f θ with small Lipschitz constant leverages this property.

Original: (ii)
Revised: (ii)

Original: The Lipschitz constant of f θ is guaranteed to be smaller than L = √ ρ .
Revised: The Lipschitz constant of f θ is guaranteed to be smaller than L = √ ρ .

Original: This provides advantages in, e.g., physical systems where Lipschitz constant guarantees translates to stability and safety assurances.
Revised: This provides advantages in, e.g., physical systems where Lipschitz constant guarantees translates to stability and safety assurances.

Original: (iii)
Revised: (iii)

Original: It leverages the intrinsic low-dimensional structure of the manifold M embedded in the ambient space.
Revised: It leverages the intrinsic low-dimensional structure of the manifold M embedded in the ambient space.

Original: In particular, this permits taking advantage of unlabeled data.
Revised: In particular, this permits taking advantage of unlabeled data.

Original: Motivations (i) and (ii) are tropes of the Lipschitz regularization literature; e.g., Oberman and Calder (2018); Finlay et al.
Revised: Motivations (i) and (ii) are tropes of the Lipschitz regularization literature; e.g., Oberman and Calder (2018); Finlay et al.

Original: (2018a)
Revised: (2018a)

Original: ; Couellan (2021); Finlay et al. (2018b); Pauli et al.
Revised: ; Couellan (2021); Finlay et al. (2018b); Pauli et al.

Original: ; Krishnan et al.
Revised: ; Krishnan et al.

Original: (2020); Shi et al.
Revised: (2020); Shi et al.

Original: ; Lindemann et al.
Revised: ; Lindemann et al.

Original: ; Arghal et al.
Revised: ; Arghal et al.

Original: Indeed, the problem formulation in (3) is inspired in similar problem formulations in which the Lipschitz constant is regularized in the ambient space, minimize
Revised: Indeed, the problem formulation in (3) is inspired in similar problem formulations in which the Lipschitz constant is regularized in the ambient space, minimize

Original: [Equation]
Revised: [Equation]

Original: subject
Revised: subject

Original: [Equation]
Revised: [Equation]

Original: A difference between (3) and (4) is that in the latter we use a Lipschitz condition that does not require differentiability.
Revised: A difference between (3) and (4) is that in the latter we use a Lipschitz condition that does not require differentiability.

Original: A more important difference is that in (4) the Lipschitz constant is regularized in the ambient space.
Revised: A more important difference is that in (4) the Lipschitz constant is regularized in the ambient space.

Original: The distance between features y and z in (4) is the
Revised: The distance between features y and z in (4) is the

Original: Euclidean distance ∥ y − z ∥ .
Revised: Euclidean distance ∥ y − z ∥ .

Original: This is disparate from the manifold metric d M ( y, z ) that is implicit in the manifold gradient constraint in (3).
Revised: This is disparate from the manifold metric d M ( y, z ) that is implicit in the manifold gradient constraint in (3).

Original: Thus, the formulation in (3) improves upon (4) because of it leverages the structure of the manifold M [cf.
Revised: Thus, the formulation in (3) improves upon (4) because of it leverages the structure of the manifold M [cf.

Original: Motivation (iii)].
Revised: Motivation (iii)].

Original: Motivations (i) and (iii) are themes of the manifold regularization literature (Belkin et al., 2005; Niyogi, 2013; Kejani et al., 2020;
Revised: Motivations (i) and (iii) are themes of the manifold regularization literature (Belkin et al., 2005; Niyogi, 2013; Kejani et al., 2020;

Original: Li et al., 2022).
Revised: Li et al., 2022).

Original: And, indeed, it is ready to conclude by invoking Green’s first identity (see Section 3.2) that the formulation in (3) is also inspired in the manifold regularization problem,
Revised: And, indeed, it is ready to conclude by invoking Green’s first identity (see Section 3.2) that the formulation in (3) is also inspired in the manifold regularization problem,

Original: [Equation]
Revised: [Equation]

Original: The difference between (3) and (5) is that in the latter the manifold Lipschitz constant is added as a regularization penalty.
Revised: The difference between (3) and (5) is that in the latter the manifold Lipschitz constant is added as a regularization penalty.

Original: This is disparate from the imposition of a manifold
Revised: This is disparate from the imposition of a manifold

Original: Lipschitz constraint in (3).
Revised: Lipschitz constraint in (3).

Original: The regularization in (5) favors solutions with small Lipschitz constant by penalizing large Lipschitz constants, while the constraint in (3) guarantees that the Lipschitz constant is bounded by ρ .
Revised: The regularization in (5) favors solutions with small Lipschitz constant by penalizing large Lipschitz constants, while the constraint in (3) guarantees that the Lipschitz constant is bounded by ρ .

Original: This is the distinction between regularizing a
Revised: This is the distinction between regularizing a

Original: Lipschitz constant versus constraining a Lipschitz constant.
Revised: Lipschitz constant versus constraining a Lipschitz constant.

Original: The constraint in (5) is also imposed at all points in the manifold, whereas the regularization in (5) is an average over the manifold.
Revised: The constraint in (5) is also imposed at all points in the manifold, whereas the regularization in (5) is an average over the manifold.

Original: Taking an average allows for large Lipschitz constants at some specific points if this is canceled out by small Lipschitz constants in other points of the manifold.
Revised: Taking an average allows for large Lipschitz constants at some specific points if this is canceled out by small Lipschitz constants in other points of the manifold.

Original: Both of these observations imply that (3) improves upon (5) because it offers global smoothness guarantees that are important in, e.g., physical systems [cf.
Revised: Both of these observations imply that (3) improves upon (5) because it offers global smoothness guarantees that are important in, e.g., physical systems [cf.

Original: Motivation (iii)].
Revised: Motivation (iii)].

Original: Remark 1 (Alternative Manifold Lipschitz Control Formulations) .
Revised: Remark 1 (Alternative Manifold Lipschitz Control Formulations) .

Original: There are three arbitrary choices in (3).
Revised: There are three arbitrary choices in (3).

Original: (a) We choose to constrain the average statistical loss E p ( x,y )
Revised: (a) We choose to constrain the average statistical loss E p ( x,y )

Original: [ ℓ ( f θ ( x ) , y )] ≤ ϵ ; (b) we choose to constrain the pointwise Lipschitz constant ∥∇ M f θ ( z ) ∥ 2 ≤ ρ ; and (c) we choose as our objective to require a target loss ϵ and minimize the Lipschitz constant L = √ ρ .
Revised: [ ℓ ( f θ ( x ) , y )] ≤ ϵ ; (b) we choose to constrain the pointwise Lipschitz constant ∥∇ M f θ ( z ) ∥ 2 ≤ ρ ; and (c) we choose as our objective to require a target loss ϵ and minimize the Lipschitz constant L = √ ρ .

Original: We can alternatively choose to constrain the pointwise loss ℓ ( f θ ( x ) , y ) ≤ ϵ , to constrain the average Lipschitz constant (cid:82) M ∥∇ M f θ ( z ) ∥ 2 p ( z ) dV ( z ) ≤ ρ [cf (5)], or to require a target smoothness L = √ ρ and minimize the loss ϵ .
Revised: We can alternatively choose to constrain the pointwise loss ℓ ( f θ ( x ) , y ) ≤ ϵ , to constrain the average Lipschitz constant (cid:82) M ∥∇ M f θ ( z ) ∥ 2 p ( z ) dV ( z ) ≤ ρ [cf (5)], or to require a target smoothness L = √ ρ and minimize the loss ϵ .

Original: All of the possible eight combinations of choices are of interest.
Revised: All of the possible eight combinations of choices are of interest.

Original: We formulate (3) because it is the most natural intersection between the regularization of Lipschitz constants in ambient spaces [cf.
Revised: We formulate (3) because it is the most natural intersection between the regularization of Lipschitz constants in ambient spaces [cf.

Original: (4)] and manifold regularization [cf.
Revised: (4)] and manifold regularization [cf.

Original: The techniques we develop in this paper can be adapted to any of the other seven alternative formulations.
Revised: The techniques we develop in this paper can be adapted to any of the other seven alternative formulations.

Original:  Learning with Global Lipschitz Constraints
Revised:  Learning with Global Lipschitz Constraints

Original: Problem (3) is a constrained learning problem that we will solve in the dual domain (Chamon and Ribeiro, 2020).
Revised: Problem (3) is a constrained learning problem that we will solve in the dual domain (Chamon and Ribeiro, 2020).

Original: To that end, observe that (3) has statistical and pointwise constraints.
Revised: To that end, observe that (3) has statistical and pointwise constraints.

Original: The loss constraint E p ( x,y )
Revised: The loss constraint E p ( x,y )

Original: [ ℓ ( f θ ( x ) , y )] ≤ ϵ is said to be statistical because it restricts the expected loss over the data distribution.
Revised: [ ℓ ( f θ ( x ) , y )] ≤ ϵ is said to be statistical because it restricts the expected loss over the data distribution.

Original: The Lipschitz constant constraints ∥∇ M f θ ( z ) ∥ 2 ≤ ρ are said to be pointwise because they are imposed for all individual points in the manifold except for a set of zero measure.
Revised: The Lipschitz constant constraints ∥∇ M f θ ( z ) ∥ 2 ≤ ρ are said to be pointwise because they are imposed for all individual points in the manifold except for a set of zero measure.

Original: Consider then a Lagrange multiplier µ associated with the statistical constraint E p ( x,y )
Revised: Consider then a Lagrange multiplier µ associated with the statistical constraint E p ( x,y )

Original: [ ℓ ( f θ ( x ) , y )] ≤ ϵ and a Lagrange multiplier distribution λ ( z ) associated with the set of pointwise constraints ∥∇ M f θ ( z ) ∥ 2 ≤ ρ .
Revised: [ ℓ ( f θ ( x ) , y )] ≤ ϵ and a Lagrange multiplier distribution λ ( z ) associated with the set of pointwise constraints ∥∇ M f θ ( z ) ∥ 2 ≤ ρ .

Original: The dual problem associated with (3) can then be written as
Revised: The dual problem associated with (3) can then be written as

Original: [Equation]
Revised: [Equation]

Original: subject to (cid:90)
Revised: subject to (cid:90)

Original: [Equation]
Revised: [Equation]

Original: We point out that in (6) we remove ρ from the Lagrangian by incorporating the dual variable constraint (cid:82) M λ ( x ) ∥∇ M f θ ( x ) ∥ 2 p ( x ) dV ( x ) = 1 ; see Appendix A for details.
Revised: We point out that in (6) we remove ρ from the Lagrangian by incorporating the dual variable constraint (cid:82) M λ ( x ) ∥∇ M f θ ( x ) ∥ 2 p ( x ) dV ( x ) = 1 ; see Appendix A for details.

Original: We henceforth use the dual problem (6) in lieu of (3).
Revised: We henceforth use the dual problem (6) in lieu of (3).

Original: Since we are interested in situations in which we do not have access to the data distribution p ( x, y ) we further consider empirical versions of (6).
Revised: Since we are interested in situations in which we do not have access to the data distribution p ( x, y ) we further consider empirical versions of (6).

Original: Consider then N i.i.d.
Revised: Consider then N i.i.d.

Original: samples ( x n , y n ) drawn from p ( x, y ) and define the empirical dual problem as,
Revised: samples ( x n , y n ) drawn from p ( x, y ) and define the empirical dual problem as,

Original: [Equation]
Revised: [Equation]

Original: where, to simplify notation, we assume no unlabeled samples are available.
Revised: where, to simplify notation, we assume no unlabeled samples are available.

Original: If unlabeled samples are given the modification is straight-forward; see Appendix B.
Revised: If unlabeled samples are given the modification is straight-forward; see Appendix B.

Original: The remainder of this section provides three technical contributions:
Revised: The remainder of this section provides three technical contributions:

Original: (C2)
Revised: (C2)

Original: To justify solutions of (7) we must show statistical consistency with respect to the primal problem (3).
Revised: To justify solutions of (7) we must show statistical consistency with respect to the primal problem (3).

Original: This is challenging because of two reasons: (i) Since we did not assume the use of a convex parameterization, (3) is not a convex problem on θ .
Revised: This is challenging because of two reasons: (i) Since we did not assume the use of a convex parameterization, (3) is not a convex problem on θ .

Original: Thus, the primal and dual problems are not necessarily equivalent.
Revised: Thus, the primal and dual problems are not necessarily equivalent.

Original: (ii) Since we are maximizing over the dual variables µ
Revised: (ii) Since we are maximizing over the dual variables µ

Original: and
Revised: and

Original: λ ( z ) , we do not know if the empirical dual formulation in (7) is close to the statistical dual formulation in (6).
Revised: λ ( z ) , we do not know if the empirical dual formulation in (7) is close to the statistical dual formulation in (6).

Original: We will overcome these two challenges and show that the empirical dual problem (7)
Revised: We will overcome these two challenges and show that the empirical dual problem (7)

Original: is a consistent
Revised: is a consistent

Original: approximation of the statistical primal problem
Revised: approximation of the statistical primal problem

Original: Proposition 1 in Section 3.1)
Revised: Proposition 1 in Section 3.1)

Original: . (C3)
Revised: . (C3)

Original: Solving (7) requires evaluating the gradient norm sum (
Revised: Solving (7) requires evaluating the gradient norm sum (

Original: [Equation]
Revised: [Equation]

Original: N )
Revised: N )

Original: [Equation]
Revised: [Equation]

Original: We
Revised: We

Original: will generalize results from the
Revised: will generalize results from the

Original: manifold regularization literature
Revised: manifold regularization literature

Original: to show that under regularity conditions on λ the gradient norm integral can be computed in a more amenable form utilizing a weighted point cloud Laplacian (Proposition 3 in Section3.
Revised: to show that under regularity conditions on λ the gradient norm integral can be computed in a more amenable form utilizing a weighted point cloud Laplacian (Proposition 3 in Section3.

Original: (C4)
Revised: (C4)

Original: We introduce a primal-dual algorithm to solve (7) (Section 3.3).
Revised: We introduce a primal-dual algorithm to solve (7) (Section 3.3).

Original: 3. Statistical consistency of the empirical dual problem
Revised: 3. Statistical consistency of the empirical dual problem

Original: We show in this section that (7) is close to (3).
Revised: We show in this section that (7) is close to (3).

Original: Doing so requires the following assumptions:
Revised: Doing so requires the following assumptions:

Original: Assumption 1.
Revised: Assumption 1.

Original: The loss ℓ is M -Lipschitz continuous, B -bounded, and convex.
Revised: The loss ℓ is M -Lipschitz continuous, B -bounded, and convex.

Original: Assumption 2.
Revised: Assumption 2.

Original: Let H =
Revised: Let H =

Original: { f θ | θ ∈ Θ ⊂ R Q } with compact Θ be the hypothesis class, and let ¯
Revised: { f θ | θ ∈ Θ ⊂ R Q } with compact Θ be the hypothesis class, and let ¯

Original: H = conv ( H ) be the closure of its convex hull.
Revised: H = conv ( H ) be the closure of its convex hull.

Original: For each ν > 0 and φ ∈ ¯ H there exists θ ∈ Θ such that simultaneously sup z ∈M | φ ( z ) − f θ ( z ) | ≤ ν and sup z ∈M ∥∇ M φ ( z ) − ∇ M f θ ( z ) ∥ ≤ ν .
Revised: For each ν > 0 and φ ∈ ¯ H there exists θ ∈ Θ such that simultaneously sup z ∈M | φ ( z ) − f θ ( z ) | ≤ ν and sup z ∈M ∥∇ M φ ( z ) − ∇ M f θ ( z ) ∥ ≤ ν .

Original: Assumption 3.
Revised: Assumption 3.

Original: The hypothesis class H is G -Lipschitz in its gradients, i.e. ∥∇ M f θ 1 ( z ) − ∇ M f θ 2 ( z ) ∥ ≤ G | f θ 1 ( z ) − f θ 2 ( z ) | .
Revised: The hypothesis class H is G -Lipschitz in its gradients, i.e. ∥∇ M f θ 1 ( z ) − ∇ M f θ 2 ( z ) ∥ ≤ G | f θ 1 ( z ) − f θ 2 ( z ) | .

Original: Assumption 4.
Revised: Assumption 4.

Original: There exists ζ ( N, δ ) ≥ 0 , and ˆ ζ ( N, δ ) ≥ 0 monotonically decreasing with N , such that
Revised: There exists ζ ( N, δ ) ≥ 0 , and ˆ ζ ( N, δ ) ≥ 0 monotonically decreasing with N , such that

Original: [Equation]
Revised: [Equation]

Original: for all θ ∈ Θ , with probability 1 − δ over independent draws ( x n , y n ) ∼ p .
Revised: for all θ ∈ Θ , with probability 1 − δ over independent draws ( x n , y n ) ∼ p .

Original: Assumption 5.
Revised: Assumption 5.

Original: There exists a feasible solution ˜ θ ∈ Θ such that E [ ℓ ( f ˜ θ ( x ) , y )] < ϵ − Mν .
Revised: There exists a feasible solution ˜ θ ∈ Θ such that E [ ℓ ( f ˜ θ ( x ) , y )] < ϵ − Mν .

Original: Assumption 1 holds for most losses utilized in practice.
Revised: Assumption 1 holds for most losses utilized in practice.

Original: Assumption 2 is a requirement on the richness of the parametrization.
Revised: Assumption 2 is a requirement on the richness of the parametrization.

Original: In the particular case of neural networks the covering constant ν is upper bounded by the universal approximation bound of the neural network.
Revised: In the particular case of neural networks the covering constant ν is upper bounded by the universal approximation bound of the neural network.

Original: Assumption 3 also holds for neural networks if we use smooth nonlinearities – e.g., hyperbolic tangents.
Revised: Assumption 3 also holds for neural networks if we use smooth nonlinearities – e.g., hyperbolic tangents.

Original: The uniform convergence property (8) is customary in learning theory to prove
Revised: The uniform convergence property (8) is customary in learning theory to prove

Original: PAC learnability and is implied by bounds on complexity measures such as VC dimension or
Revised: PAC learnability and is implied by bounds on complexity measures such as VC dimension or

Original: Rademacher complexity (Mohri et al., 2018; Vapnik, 1999; Shalev-Shwartz and Ben-David, 2014).
Revised: Rademacher complexity (Mohri et al., 2018; Vapnik, 1999; Shalev-Shwartz and Ben-David, 2014).

Original: The following theorem provides the desired bound.
Revised: The following theorem provides the desired bound.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let ˆ µ ⋆ ,
Revised: Let ˆ µ ⋆ ,

Original: ˆ λ ⋆ be solutions of the empirical dual problem (7) .
Revised: ˆ λ ⋆ be solutions of the empirical dual problem (7) .

Original: Under assumptions 1–5, there exists ˆ θ ⋆ ∈ argmin θ ˆ L ( θ, ˆ µ ⋆ , ˆ λ ⋆ ) such that, with probability 1 − 5 δ ,
Revised: Under assumptions 1–5, there exists ˆ θ ⋆ ∈ argmin θ ˆ L ( θ, ˆ µ ⋆ , ˆ λ ⋆ ) such that, with probability 1 − 5 δ ,

Original: [Equation]
Revised: [Equation]

Original: where ∆ = max(ˆ µ ⋆ , µ ⋆ ) ≤ C for a constant C < ∞ , where µ ⋆ , ˆ µ ⋆ are solutions of (6) , and (7) respectively.
Revised: where ∆ = max(ˆ µ ⋆ , µ ⋆ ) ≤ C for a constant C < ∞ , where µ ⋆ , ˆ µ ⋆ are solutions of (6) , and (7) respectively.

Original: Proposition 1 shows that the empirical dual problem 7 is statistically consistent.
Revised: Proposition 1 shows that the empirical dual problem 7 is statistically consistent.

Original: That is to say, for any realization of N according to p , the difference between the empirical dual problem (7) and the statistical smooth learning problem (3) decreases as N increases.
Revised: That is to say, for any realization of N according to p , the difference between the empirical dual problem (7) and the statistical smooth learning problem (3) decreases as N increases.

Original: This difference is bounded in terms of the richness of the parametrization ( ν ), the difficulty of the fit requirement (as expressed by the optimal dual variables µ ⋆ , ˆ µ ⋆ ), and the number of samples ( N ).
Revised: This difference is bounded in terms of the richness of the parametrization ( ν ), the difficulty of the fit requirement (as expressed by the optimal dual variables µ ⋆ , ˆ µ ⋆ ), and the number of samples ( N ).

Original: The guarantee has a form typical for constrained learning problems (Chamon and Ribeiro, 2020).
Revised: The guarantee has a form typical for constrained learning problems (Chamon and Ribeiro, 2020).

Original: Proposition 1 states that we are able to predict what is the minimum norm of the gradient that a function class can have while satisfying an expected ϵ loss.
Revised: Proposition 1 states that we are able to predict what is the minimum norm of the gradient that a function class can have while satisfying an expected ϵ loss.

Original: This is important, because we do not require access to the distribution of the samples p , only a set of N samples sampled according to this distribution.
Revised: This is important, because we do not require access to the distribution of the samples p , only a set of N samples sampled according to this distribution.

Original: On the other hand, Propositiondoes not state that by solving the dual problem 7 we will obtain a solution of the primal problem 3.
Revised: On the other hand, Propositiondoes not state that by solving the dual problem 7 we will obtain a solution of the primal problem 3.

Original: The following proposition provides a bound on the near feasibility of the solution of 7 with respect to the solution of 3.
Revised: The following proposition provides a bound on the near feasibility of the solution of 7 with respect to the solution of 3.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let ˆ µ ⋆ ,
Revised: Let ˆ µ ⋆ ,

Original: ˆ λ ⋆ be solutions of the empirical dual problem (7) .
Revised: ˆ λ ⋆ be solutions of the empirical dual problem (7) .

Original: Under assumptions 1–5, there exists ˆ θ ⋆ ∈ argmin θ ˆ L ( θ, ˆ µ ⋆ , ˆ λ ⋆ ) such that, with probability 1 − 5 δ ,
Revised: Under assumptions 1–5, there exists ˆ θ ⋆ ∈ argmin θ ˆ L ( θ, ˆ µ ⋆ , ˆ λ ⋆ ) such that, with probability 1 − 5 δ ,

Original: [Equation]
Revised: [Equation]

Original: Proposition 2 provides near optimality, and near feasibility conditions for solutions ˆ θ ∗ obtained through the empirical dual problem (7).
Revised: Proposition 2 provides near optimality, and near feasibility conditions for solutions ˆ θ ∗ obtained through the empirical dual problem (7).

Original: The difference between the maximum gradient of the obtained solution θ and the optimal value P ∗ is bounded by the number of samples N , as well as the empirical constraint violation.
Revised: The difference between the maximum gradient of the obtained solution θ and the optimal value P ∗ is bounded by the number of samples N , as well as the empirical constraint violation.

Original: Notice that even though the optimal dual variable ˆ µ is not known, the constraint violation can be evaluated in practice, as it only requires to evaluate the obtained function over the N given samples.
Revised: Notice that even though the optimal dual variable ˆ µ is not known, the constraint violation can be evaluated in practice, as it only requires to evaluate the obtained function over the N given samples.

Original: Remark 2 (Interpolators) .
Revised: Remark 2 (Interpolators) .

Original: In practice, the number of parameters in a parametric function (e.g. Neural Network) tends to exceed the dimension of the input, which allows functions to interpolate the data, i.e. to attain zero loss on the dataset.
Revised: In practice, the number of parameters in a parametric function (e.g. Neural Network) tends to exceed the dimension of the input, which allows functions to interpolate the data, i.e. to attain zero loss on the dataset.

Original: Proposition 2, presents a connection to interpolating functions.
Revised: Proposition 2, presents a connection to interpolating functions.

Original: By setting ϵ = 0 , if the function achieves zero error over the empirical distribution i.e. ℓ ( f ˆ θ ∗ ( x n ) , y n ) = 0 , for all n ∈ [ N ] , then the dependency on µ ∗ disappears.
Revised: By setting ϵ = 0 , if the function achieves zero error over the empirical distribution i.e. ℓ ( f ˆ θ ∗ ( x n ) , y n ) = 0 , for all n ∈ [ N ] , then the dependency on µ ∗ disappears.

Original: This implies that over interpolating classifiers, the one with the minimum
Revised: This implies that over interpolating classifiers, the one with the minimum

Original: Lipschitz constant over the samples, is probably the one that attains the minimum Lipschitz constant over the whole manifold.
Revised: Lipschitz constant over the samples, is probably the one that attains the minimum Lipschitz constant over the whole manifold.

Original: 3. From Manifold Gradient to Discrete Laplacian
Revised: 3. From Manifold Gradient to Discrete Laplacian

Original: We derive an alternative way of computing the integral of the norm of the gradient utilizing samples.
Revised: We derive an alternative way of computing the integral of the norm of the gradient utilizing samples.

Original: To do so, we define the normalized point cloud Laplacian according a probability distribution λ .
Revised: To do so, we define the normalized point cloud Laplacian according a probability distribution λ .

Original: Definition 2 (Point-cloud Laplacian) .
Revised: Definition 2 (Point-cloud Laplacian) .

Original: Consider a set of points x 1 , . .
Revised: Consider a set of points x 1 , . .

Original: , x N ∈ M , sampled according to probability λ : M → R .
Revised: , x N ∈ M , sampled according to probability λ : M → R .

Original: The normalized graph Laplacian of f θ at z ∈ M is defined as
Revised: The normalized graph Laplacian of f θ at z ∈ M is defined as

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: As long as the function considered is smooth enough, the following convergence result holds:
Revised: As long as the function considered is smooth enough, the following convergence result holds:

Original: Proposition 3 (Point Cloud Estimate) .
Revised: Proposition 3 (Point Cloud Estimate) .

Original: Let Λ be the set of probability distributions defined a compact d -dimensional differentiable manifold M isometrically embedded in R N such that Λ = { λ : 0 < a ≤ λ ( z ) ≤ b < ∞ , | ∂λ∂x | ≤ c < ∞ and | ∂ 2 λ ∂x 2 | ≤ d < ∞ for all z ∈ M} , and let f θ , with θ ∈ Θ , be a family of functions with uniformly bounded derivatives up to ordervanishing at the boundary ∂ M .
Revised: Let Λ be the set of probability distributions defined a compact d -dimensional differentiable manifold M isometrically embedded in R D such that Λ = { λ : 0 < a ≤ λ ( z ) ≤ b < ∞ , | ∂λ∂x | ≤ c < ∞ and | ∂ 2 λ ∂x 2 | ≤ d < ∞ for all z ∈ M} , and let f θ , with θ ∈ Θ , be a family of functions with uniformly bounded derivatives up to ordervanishing at the boundary ∂ M .

Original: For any ϵ > 0 , δ > 0 , for all N > N 0 ,
Revised: For any ϵ > 0 , δ > 0 , for all N > N 0 ,

Original: [Equation]
Revised: [Equation]

Original: where the point cloud laplacian L t λ,N f θ is as defined in 2, with t = N 
Revised: where the point cloud laplacian L t λ,N f θ is as defined in 2, with t = N −

Original: [Equation]
Revised: [Equation]

Original: for any α > 0 .
Revised: for any α > 0 .

Revised: The proof of Proposition 3 relies on two steps.

Revised: First, we relate the integral of the norm of the gradient over the manifold with the integral of the continuous Laplace-Beltrami operator by virtue of Green’s identity.

Revised: Second, we approximate the value of the Laplace-Beltrami operator by the point-cloud Laplacian.

Original: Proposition 3 connects the integral of the norm of the gradient of function f
Revised: Proposition 3 connects the integral of the norm of the gradient of function f

Original: θ with a point cloud Laplacian operator. This result connects the dual problem 7, with the primal 3 while allowing for a more amenable way of computing the integral.
Revised: θ with a point cloud Laplacian operator, this result connects the dual problem 7, with the primal 3 while allowing for a more amenable way of computing the integral.

Original: Remark 3 (Laplacian Regularization and Manifold Lipschitz) .
Revised: Remark 3 (Laplacian Regularization and Manifold Lipschitz) .

Original: The dual problem (6) is closely related to the manifold regularization problem (5) .
Revised: The dual problem (6) is closely related to the manifold regularization problem (5) .

Original: In particular, the two become equivalent by substituting ρ = µ − 1 , and utilizing a uniform distribution for λ .
Revised: In particular, the two become equivalent by substituting ρ = µ − 1 , and utilizing a uniform distribution for λ .

Original: The key difference between the problems is given by the dual variable λ , which can be though as a probability distribution over the manifold that penalizes regions of the manifold where the norm of the gradient of f θ is larger.
Revised: The key difference between the problems is given by the dual variable λ , which can be though as a probability distribution over the manifold that penalizes regions of the manifold where the norm of the gradient of f θ is larger.

Original: The standard procedure in Laplacian regularization is to calculate the graph-Laplacian of the set of points L utilizing the heat kernel, and compute the integral by f T Lf , where f = [ f θ ( x 1 ) , . . .
Revised: The standard procedure in Laplacian regularization is to calculate the graph-Laplacian of the set of points L utilizing the heat kernel, and compute the integral by f T Lf , where f = [ f θ ( x 1 ) , . . .

Original: , f θ ( x n )] T .
Revised: , f θ ( x n )] T .

Original: In the case of Manifold Lipschitz, the same product can be computed, but utilizing the re-weighted point cloud laplacian.
Revised: In the case of Manifold Lipschitz, the same product can be computed, but utilizing the re-weighted point cloud laplacian.

Original: 3. Dual Ascent Algorithm
Revised: 3. Dual Ascent Algorithm

Original: We outline an iterative and empirical primal-dual algorithm to solve the dual problem.
Revised: We outline an iterative and empirical primal-dual algorithm to solve the dual problem.

Original: Upon the initialization of θ 0 , and λ 0 , µ 0 , we set to minimize the dual function as follows,
Revised: Upon the initialization of θ 0 , and λ 0 , µ 0 , we set to minimize the dual function as follows,

Original: [Equation]
Revised: [Equation]

Original: where η θ is a positive step-size.
Revised: where η θ is a positive step-size.

Original: Note that to solve problem (16), we can either utilize the gradient version the the lagrangian (cf.
Revised: Note that to solve problem (16), we can either utilize the gradient version the the lagrangian (cf.

Original: equation (7)), or the point-cloud Laplacian (cf.
Revised: equation (7)), or the point-cloud Laplacian (cf.

Original: equation (15)).
Revised: equation (15)).

Original: Consequent to updating θ , we update the dual variables as follows,
Revised: Consequent to updating θ , we update the dual variables as follows,

Original: [Equation]
Revised: [Equation]

Original: n )

Original: ∥ 2 for all data points

Original: [Equation]

Original: n
Revised: N

Original: [Equation]
Revised: N (cid:88) n = ℓ ( f θ ( x n ) , y n ) − ϵ (cid:19)(cid:21) + , (17) ˜ λ k +1 ( x n ) = λ k ( x n ) + η λ ∥∇ M f θ ( x n ) ∥ 2 for all data points x n (18) λ

Original:  = argmin [Equation]
Revised: k +1 = argmin λ

Original: ∥ 
Revised: ∥ ˜ λ

Original: [Equation] ∥ , such that
Revised: k +1 − λ k +1 ∥ , such that

Original: [Equation]
Revised: N (cid:88) n = λ k +1 ( x n ) = N.

Original:  where η µ , η λ are a positive step-sizes.
Revised: (19) where η µ , η λ are a positive step-sizes.

Original: Note that we require a convex projection over ˜ λ to satisfy the normalizing constraint.
Revised: Note that we require a convex projection over ˜ λ to satisfy the normalizing constraint.

Original: In step (18) we require to estimate the norm of the gradient at data point x n , which we estimate for the neighboring data points.
Revised: In step (18) we require to estimate the norm of the gradient at data point x n , which we estimate for the neighboring data points.

Original: Intuitively, the primal dual procedure increases the value of λ ( x n ) at points in which the norm of the gradient is larger.
Revised: Intuitively, the primal dual procedure increases the value of λ ( x n ) at points in which the norm of the gradient is larger.

Original: The role of µ is to enforce the loss ℓ to be smaller than ϵ , a larger value of µ would increase the relative importance over the norm of the integral.
Revised: The role of µ is to enforce the loss ℓ to be smaller than ϵ , a larger value of µ would increase the relative importance over the norm of the integral.

Original: See Appendix F.
Revised: See Appendix F.

Original:  Numerical Results 4. Navigation Controls Problem
Revised:  Numerical Results 4. Navigation Controls Problem

Original: In this section, we consider the problem of continuous navigation of an agent. 
Revised: In this section, we consider the problem of continuous navigation of an agent. Method Trajectories ERM

Original:  The agent’s objective is to reach a goal while avoiding obstacles.
Revised: Ambient Reg. Manifold Reg. Manifold Lipschitz The agent’s objective is to reach a goal while avoiding obstacles.

Original: The state space of the agent is given by its position, and navigates by taking actions on the velocity.Method
Revised: The state space of the agent is given by its position, and navigates by taking actions on the velocity.

Original: [Table] We construct a square grid of points in the free space i.e. outside of the obstacles, and find the shortest path between two starting positions and the goal along the grid.
Revised: We construct a square grid of points in the free space i.e. outside of the obstacles, and find the shortest path between two starting positions and the goal along the grid.

Original: For those two grid trajectories, we compute the optimal actions to be taken at each point in order to follow the trajectory.
Revised: For those two grid trajectories, we compute the optimal actions to be taken at each point in order to follow the trajectory.

Original: The learner is equipped with both labeled trajectories, as well as the unlabeled point grid.
Revised: The learner is equipped with both labeled trajectories, as well as the unlabeled point grid.

Original: To leverage the manifoldstructure of the data, we consider the grid of points, and we construct the point cloud Laplacian considering adjacent points in the grid. As measure of merit, we take 100 random points and we compute the trajectories. A trajectory is successful if it achieves the goal without colliding. The results are shown in Table 1, and the learned functions in Figure 2.
Revised: To leverage the manifold

Original: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (a) Dataset 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (b) Manifold Lipschitz.
Revised: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (a) Dataset 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (b) Manifold Lipschitz.

Original: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (c) Dual variables λ .
Revised: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (c) Dual variables λ .

Original: [Figure] ) ERM.
Revised: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (d) ERM.

Original: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (e) Ambient Regularization.
Revised: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (e) Ambient Regularization.

Original: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (f) Manifold Regularization.
Revised: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0     (f) Manifold Regularization.

Revised: structure of the data, we consider the grid of points, and we construct the point cloud Laplacian considering adjacent points in the grid.

Revised: As measure of merit, we take 100 random points and we compute the trajectories.

Revised: A trajectory is successful if it achieves the goal without colliding.

Revised: The results are shown in Table 1, and the learned functions in Figure 2.

Original: 4. Ground Robot Error Prediction
Revised: 4. Ground Robot Error Prediction

Revised: Method

Revised: Grass

Revised: Pavement

Revised: ERM 0 .0 .

Original:  In this real world experiment, we seek to learn the error that a model would make in predicting the dynamics of a ground robot Koppel et al.
Revised: Ambient Reg. Manifold Reg. Manifold Lipschitz 0 .0 . In this real world experiment, we seek to learn the error that a model would make in predicting the dynamics of a ground robot Koppel et al.

Original: We posses trajectories, in the form of time series of the control signals of an iRobot Packbot making turns on both pavement, and grass.
Revised: We posses trajectories, in the form of time series of the control signals of an iRobot Packbot making turns on both pavement, and grass.

Original: For each trajectory, the mismatch between the real and the model-predicted states is quantified.
Revised: For each trajectory, the mismatch between the real and the model-predicted states is quantified.

Original: Given a dataset of trajectories and errors made by the model, the objective is to learn the error that a model will make.
Revised: Given a dataset of trajectories and errors made by the model, the objective is to learn the error that a model will make.

Original: Details of the experiment can be found in G.3.
Revised: Details of the experiment can be found in G.3.

Original: Method

Original: Grass

Original: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we presented a constrain learning method to obtain smooth functions over manifold data.
Revised: In this work, we presented a constrain learning method to obtain smooth functions over manifold data.

Original: We have shown that under mild conditions, the problem of finding smooth functions over a manifold can be reformulated as a weighted point cloud Laplacian penalty over varying probability distributions whose dynamics are govern by the constraint violations.
Revised: We have shown that under mild conditions, the problem of finding smooth functions over a manifold can be reformulated as a weighted point cloud Laplacian penalty over varying probability distributions whose dynamics are govern by the constraint violations.

Original: Three numerical examples validate the empirical advantages of obtaining functions that vary smoothly over the data.
Revised: Three numerical examples validate the empirical advantages of obtaining functions that vary smoothly over the data.