L IPSCHITZ REGULARIZED D EEP N EURAL N ETWORKS CONVERGE AND GENERALIZE
Anonymous authors
Paper under double-blind review
A BSTRACT
Generalization of deep neural networks (DNNs) is an open problem which, if solved, could impact the reliability of deep neural network architectures.
In this paper, we show that if the usual ﬁdelity term used in training DNNs is augmented by a Lipschitz regularization term, then the networks converge and generalize.
The convergence is in the limit as the number of data points, n → ∞ , while also allowing the network to grow as needed to ﬁt the data.
Two regimes are identiﬁed:
in the case of clean labels, we prove convergence to the label function which corresponds to zero loss, in the case of corrupted labels we prove convergence to a regularized label function which is the solution of a limiting variational problem.
In both cases, a convergence rate is also provided.
I NTRODUCTION
While deep neural networks networks (DNNs) give more accurate predictions than other machine learning methods (LeCun et al., 2015), they lack some of the performance guarantees of these methods.
One step towards performance guarantees for DNNs is a proof of convergence with a rate, which could lead to quantitative error estimates.
In this paper, we present such a result, for Lipschitz regularized DNNs.
A proof of generalization follows as a consequence of convergence.
We begin by establishing the notation for our problem.
We consider the classiﬁcation problem to ﬁx ideas, although regularization can apply to other problems as well.
Deﬁnition 1.1.
Assume the data is normalized so that the data space is X = [0 , 1] d .
Write D n = x 1 , . . .
, x n for the training data.
Assume D n is a sequence of i.i.d.
random variables on X sampled from the probability distribution ρ .
We consider the classiﬁcation problem with m labels which are embedded into the probability simplex, the label space, Y ⊂ R D .
Write u 0 : X → Y for the map from data to label space, so that y i =
u 0 ( x i ) .
Our results point towards improved generalization results using Lipschitz regularization, which we deﬁne now.
Deﬁnition 1.2.
Choose norms (cid:107) · (cid:107) Y , and (cid:107) · (cid:107) X on X and Y , respectively.
The Lipschitz constant (in these norms) of a function u : X 0 ⊂ X
→ Y is given by
[Equation]
When X 0 is all of X , we write Lip( u ; X ) =
Lip( u ) .
The Lipschitz constant of the data is
Lip( u 0 ; D n ) .
Write u ( x ; w ) for the map from the input to data to the last layer of the network.
1 We consider the variational problem with Lipschitz regularization
[Equation]
The ﬁrst term in (1) is the usual averaged loss on the training data D n .
The second term in (1) the
Lipschitz regularization term: the excess Lipschitz constant of the map u , compared to the constant
L 0 .
Choosing λ > 0 in (1) introduces a Lipschitz regularization penalty to the standard learning problem, which corresponds to λ = 0 .
In theory we take L 0 =
Lip( u 0 ) , the Lipschitz constant of the data on the whole data manifold (discussed below).
Remark 1.3 .
In practice, Lip( u 0 ) can be estimated by the Lipschitz constant of the data,
Lip( u 0 ; D n ) .
In fact, for many common data sets, the Lipschitz constant is very small and the
Lipschitz constants of networks is much larger (Finlay & Oberman, 2018), so in practice we can set
L 0 = 0 .
However for clean data, if we wish to recover u 0 exactly, we need L 0 > 0 .
If u is differentiable 2 , then the boundedness of X implies
[Equation]
where (cid:107) M (cid:107) X,Y is the matrix norm induced by the norms on X and Y .
In practise, we take the ∞ -norm for Y and the 2-norm for X , and use explicit formulas for the (cid:107) M (cid:107) ∞ , 2 norm, see § B. The
Lipschitz constant of u , Lip( u ; X ) , can thus be estimated from below by the maximum of the norm of the gradient on a minibatch
[Equation]
Our analysis will apply to the problem (1) which is convex in u , and does not depend explicitly on the weights, w .
Of course, once u is restricted to a ﬁxed neural network architecture, the corresponding minimization problem becomes non-convex in the weights.
Our analysis can avoid the dependence on the weights because we make the assumption that there are enough parameters so that u can exactly ﬁt the training data.
The assumption is justiﬁed by Zhang et al.
As we send n → ∞ for convergence, we require that the network also grow, in order to continue to satisfy this assumption.
Our results apply to other non-parametric methods in this regime.
In Figure 1 we illustrate the solution of (1) (with L 0 = 0 ), using synthetic one dimensional data.
In this case, the labels {− 1 , 0 , 1 } are embedded naturally into Y = R , and λ = 0 .
Notice that the solution matches the labels exactly on a subset of the data.
In the second part of the ﬁgure, we show a solution with corrupted labels which introduce a large Lipschitz constant, in this case, the solution reduces the Lipschitz constant, thereby correcting the errors.
1. R ELATED WORK AND APPLICATIONS
Generalization bounds have been obtained previously via VC dimension analysis of neural networks (Bartlett, 1997). The generalization rates have factors of the form A k for a k -layer neural network with bounds (cid:107) w i (cid:107) ≤ A for all weight vectors w i in the network. Such bounds are only applicable for low-complexity networks.
Other works have considered connections between generalization and stability (Bousquet & Elisseeff, 2002; Xu & Mannor, 2012).
More recently, (Bartlett et al., 2017) proposed the Lipschitz constant of the network as a candidate measure for the Rademacher complexity, which is a measure of generalization (Shalev-Shwartz & Ben-David, 2014, Chapter 26).
Also, Cranko et al.
(2018) showed that Lipschitz regularization can be viewed as a special case of distributional robustness.
Unlike other recent contributions such as (Hardt et al., 2015), our analysis does not depend on the training method.
In fact, our analysis has more in common with inverse problems in image processing, such as Total Variation denoising and inpainting (Bertalmio et al., 2000; Rudin et al., 1992).
For further discussion, see Appendix C.
The estimate of Lip( u ; X ) provided by (2) can be quite different from the the Tychonoff gradient regularization (Drucker & Le Cun, 1992),
[Equation]
since (2) corresponds to a maximum of the values of the norms, and the previous equation corresponds to average of the values.
In fact, recent work on semi-supervised learning suggests that higher p -norms of the gradient are needed for generalization when the data manifold is not well approximated by the data (El Alaoui et al., 2016; Calder, 2017; Kyng et al., 2015; Slepcev & Thorpe, 2017).
In Figure 2 we compare to the problems in Figure 1 using Tychonoff regularization.
The Tychonoff regularization is less effective at correcting errors.
The effect is more pronounced in higher dimensions.
An upper bound for the Lipschitz constant of the model is given by the norm of the product of the weight matrices (Szegedy et al., 2013, Section 4.3).
Let w
=
w 1 , . . . , w J ) be the weight matrices for each layer.
Then
[Equation]
Regularization of the network using methods based on (3) has been implemented recently in (Gouk et al., 2018) and (Yoshida & Miyato, 2017).
Because the upper bound in (3) does not take into account the coefﬁcients in weight matrices which are zero due to the activation functions, the gap in the inequality can be off by factors of many orders of magnitude for deep networks (Finlay &
Oberman, 2018).
Implementing (2) can be accomplished using backpropagation in the x variable on each label, which can become costly for m large.
Special architectures could also be used to implement Lipschitz regularization, for example, on a restricted architecture, Liao et al.
(2018) renormalized the weight matrices of each layer to be norm 1.
In practise, the computational cost can be reduced by regular- izing the loss applied to the model (cid:96) ◦ u , instead
[Equation]
which requires only one backpropagation.
Lipschitz regularization may help with adversarial examples (Szegedy et al., 2013)
(Goodfellow et al., 2014) which poses a problem for model reliability (Goodfellow et al., 2018).
Since the Lipschitz constant L (cid:96) of the loss, (cid:96) , controls the norm of a perturbation
[Equation]
maps with smaller Lipschitz constants may be more robust to adversarial examples.
Finlay & Oberman (2018) implemented Lipschitz regularization of the loss, as in (4), and achieved better robustness against adversarial examples, compared to adversarial training (Goodfellow et al., 2014) alone.
Lipschitz regularization may also improve stability of GANs.
1-Lipschitz networks are also important for Wasserstein-GANs (Arjovsky et al., 2017)
(Arjovsky & Bottou, 2017).
In (Wei et al., 2018)
the gradient penalty away from norm 1 is implemented, augmented by a penalty around perturbed points, with the goal of improved stability.
Spectral regularization for GANs was implemented in (Miyato et al., 2018).
Problem 1.4 .
Consider the following problem inspired by (Zhang et al., 2016).
Suppose we are given a labelled data set, which has Lipschitz constant Lip( D )
=
O (1) , where
[Equation]
Consider making copies of 10 percent of the data, adding a vector of norm (cid:15) to the perturbed data points, and changing the label.
Call the new dirty data set ˜ D .
The new data set has Lip( ˜ D ) =
O (1 /(cid:15) ) .
However, the distribution of the values on the right hand side of (5) are mostly below
Lip( D ) with a small fraction O (1 /(cid:15) ) , since the duplicated images are (cid:15) apart but with different labels.
L IPSCHITZ REGULARIZATION AND CONVERGENCE 2. L IMITING PROBLEM
The variational problem (1) admits Lipschitz continuous minimizers, but in general the minimizers are not unique.
When L 0 =
Lip( u 0 ) , it is clear that u 0 , is a solution of (1): both the loss term and the regularization term are zero when applied to u 0 .
In addition, any L 0 -Lipschitz extension of u 0 | D n is also a minimizer of (1), so solutions are not unique.
Let u n be any solution of the Lipschitz regularized variational problem (1).
We study the limit of u n as n → ∞ .
Since the empirical probability measures ρ n converge to the data distribution ρ , the continuum variational problem corresponding to (1) is
[Equation]
where in (6) we have introduced the following notation.
Deﬁnition 2.1.
Given the loss function, (cid:96) , a map u : X → Y , and a probability measure, µ , supported on X , deﬁne
[Equation]
to be the expectation of the loss with respect to the measure.
In particular, the generalization loss of the map u : X
→ Y is given by L [ u, ρ ] .
Write L [ u, D n ] :
= L [ u, ρ n ] for the average loss on the data set D n , where ρ n := 1 n (cid:80)
δ x
i is the empirical measure corresponding to D n .
Remark 2.2 .
Generalization is deﬁned in (Goodfellow et al., 2016, Section 5.2) as the expected value of the loss function on a new input sampled from the data distribution.
As deﬁned, the full generalization error includes the training data, but it is of measure zero, so removing it does not change the value.
2. L OSS FUNCTION ASSUMPTIONS
We introduce the following assumption on the loss function.
Assumption 2.3 (Loss function) .
The function (cid:96) : Y × Y → R is a loss function if it satisﬁes (i) (cid:96) ≥ 0 , (ii) (cid:96) ( y 1 , y 2 )
=
if and only if y 1 = y 2 , and (iii) (cid:96) is strictly convex in y 1 .
Example 2.4 ( R D with L 2 loss) .
Set Y =
R D , and let each label be a basis vector.
Set (cid:96) ( y 1 , y 2 )
= (cid:107) y 1 − y 2 (cid:107) 22 to be the L 2 loss.
Example 2.5 (Classiﬁcation) .
In classiﬁcation problems, the output of the network is a probability vector on the labels.
Thus Y = ∆ p , the p -dimensional probability simplex, and each label is mapped to a basis vector.
The cross-entropy loss is given by (cid:96) KL ( y, z )
= − (cid:80) pi
=
z i log( y i /z i ) .
For labels, (cid:96) KL ( y, e k )
= − log( y k ) .
Example 2.6 .
In the classiﬁcation setting, it is often the case that the softmax function
[Equation]
is combined with the cross-entropy loss.
In this paper, we choose to absorb softmax into the last layer of the DNN, and assume the output u ( x ) of the network lies in the probability simplex.
If the output, z , of the second to last layer of the DNN, which is the input to softmax in (7), lies in a compact set, i.e., | z j | ≤ C for all i and some C > 0 , then softmax ( z ) j ≥ e − 2 C , and so the range of softmax lies in the set
[Equation]
and y 1 + · · · + y D = 1 } , which is strictly interior to the probability simplex.
Restricted to A , the cross-entropy loss (cid:96) KL is strongly convex and Lipschitz continuous, which is required in Theorems 2.11 and 2.15 below.
In our analysis, it is slightly more convenient to deﬁne the regularized cross entropy loss with parameter (cid:15) > 0 by
[Equation]
For classiﬁcation problems, where z = e k , we have (cid:96) KL(cid:15)
( y, e k ) =
− log( y k + (cid:15) ) , which is Lipschitz and strongly convex for any 0 ≤ y i ≤ 1 within the probability simplex.
Thus, the regularized cross entropy (cid:96) KL(cid:15) satisﬁes the strong convexity and Lipschitz regularity required by Theorems 2.11 and 2.15 on the whole probability simplex.
We note that while (cid:96) KL(cid:15) ( e k , e k ) =
− log(1 + (cid:15) )
(cid:54) = 0 , as is required by Assumption 2.3 (ii), it is trivial to apply the theory by adding the constant C =
log(1+ (cid:15) ) to the loss.
2. C ONVERGENCE RESULT
Here, we show that solutions of the random variational problem (1) converge to solutions of (6).
We make the standard manifold assumption (Chapelle et al., 2006), and assume the data distribution ρ is a probability density supported on a compact, smooth, m -dimensional manifold M embedded in X =
[0 , 1] d , where m (cid:28) d .
We denote the probability density again by ρ : M → [0 , ∞ ) .
Hence, the data D n is a sequence x 1 , . . .
, x n of i.i.d.
random variables on M with probability density ρ .
Associated with the random sample we have the closet point projection map σ n : X → { x 1 , . . .
, x n } ⊂ X that satisﬁes
[Equation]
for all x ∈ X .
We recall that W 1 , ∞ ( X ; Y ) is the space of Lipschitz mappings from X to Y .
Throughout this section, C, c > 0 denote positive constants depending only on M , and we assume
C ≥ 1 and 0 < c < 1 .
We follow the analysis tradition of allowing the particular values of C and c to change from line to line.
We establish that that minimizers of (6) are unique on M , which follows from the strict convexity of the loss restricted to the data manifold M , in Theorem A.1.
See also Figure 3 which shows how the solutions need not be unique off the data manifold.
Our ﬁrst convergence result is in the case where Lip[ u 0 ] ≤ L 0 , and so the Lipschitz regularizer is not fully active.
This corresponds to the case of clean labels.
We state our result in generality, for approximate minimizers of (1), and specialize to the case Lip[ u 0 ] ≤ L 0 in Remark 2.8.
Theorem 2.7 (Convergence result) .
Assume inf x ∈M ρ ( x ) > 0 .
For any t > 0 , with probability at least 1 − Ct − 1 n − ( ct − 1) every sequence u n ∈ W 1 , ∞ ( X ; Y ) with zero empirical loss L [ u 0 , ρ n ] =satisﬁes
[Equation]
Remark 2.8 .
If u n ∈ W 1 , ∞ ( X ; Y ) is any sequence of minimizers of (1) and Lip[ u 0 ] ≤ L 0 , then
J [ u n ] ≤ J [ u 0 ] = 0 .
Thus, Lip[ u n ] ≤ L 0 and Theorem 2.7 applies to the sequence u n , yielding
[Equation]
It is important to note that Theorem 2.7 does not requires u n to be minimizers of (1)—we just require zero empirical loss, which is often achieved in practice (Zhang et al., 2016).
This allows for approximation errors in solving (1) on the whole domain X , due to the restriction that u must be expressed via a Deep Neural Network.
As an immediate corollary, we can prove that the generalization loss converges to zero, and so we obtain generalization.
Corollary 2.9.
Assume that for some q ≥ 1 the loss (cid:96) satisﬁes
[Equation]
Then under the assumptions of Theorem 2.
[Equation]
holds with probability at least 1 − Ct − 1 n − ( ct − 1) .
Proof.
By (8), we can bound the generalization loss as follows
[Equation]
The proof is completed by invoking Theorem 2.7.
We now turn to the proof of Theorem 2.7, which requires a bound on the distance between the closest point projection σ n and the identity.
The result is standard in probability, and we include it for completeness in Lemma 2.10 proved in § A.1.
We refer the interested reader to (Penrose et al., 2003) for more details.
Lemma 2.10.
Suppose that inf M ρ > 0 .
Then for any t >
[Equation]
with probability at least 1 − Ct − 1 n − ( ct − 1) .
We now give the proof of Theorem 2.7.
Proof of Theorem 2.7.
Since L [ u n , ρ n ] =
we have u 0 ( x i ) =
u n ( x i ) for all 1 ≤ i ≤ n .
Thus for any x ∈ X we have
[Equation]
Y ≤ (cid:107) u 0 ( x ) − u 0 ( σ n ( x ))
(cid:107) Y + (cid:107) u n ( σ n ( x ))
− u n ( x ) (cid:107) Y ≤ ( L 0 + Lip[ u n ]) (cid:107) x − σ n ( x ) (cid:107) X .
Therefore, we deduce
[Equation]
The proof is completed by invoking Lemma 2.10.
2. C ONVERGENCE FOR NOISY LABELS
We now address problems with some degree of uncertainty.
Recall the discussion in Problem 1.4 of the setting where labels are corrupted, which is also illustrated in the right hand side of Figure 1.
We assume the true labels u true are corrupted by additive error of the form
[Equation]
for an error function e : X → Y .
We assume the true labels u true have a small Lipschitz constant
L 0 , while the error vector e has a much larger Lipschitz constant, leading to Lip( u 0 )
(cid:29) L 0 .
A simple model is described in Problem 1.4.
In this case, it is desirable to ﬁt some, but not all, of the data.
In particular, we wish to ﬁt the subset of the labels that generates the low Lipschitz constant L 0 , while hopefully ignoring the rest, which correspond to pairs of labels that generate a high Lipschitz constant.
In this setting, the Lipschitz regularizer is active, leading to smoothing of the noisy labels as depicted on the right side of Figure 1.
Our main results (Theorems 2.11 and 2.15) show that minimizers of J n converge to minimizers of J almost surely as the number of training points n tends to ∞ .
Theorem 2.11.
Suppose that inf M ρ > 0 , (cid:96) : Y × Y → R is Lipschitz, and let u ∗ ∈ W 1 , ∞ ( X ; Y ) be any minimizer of (6) .
Then with probability one u n −→ u ∗ uniformly on M as n → ∞ , (9) where u n is any sequence of minimizers of (1) .
Furthermore, every uniformly convergent subsequence of u n converges on X to a minimizer of (6) .
Remark 2.12 .
In Theorem 2.11 and Theorem 2.15 below, the sequence u n does not, in general, converge on the whole domain X .
The important point is that the sequence converges on the data manifold M , and solves the variational problem (6) off of the manifold, which ensures that the output of the DNN is stable with respect to the input.
See Figure 3.
The proof of Theorem 2.11 requires a preliminary Lemma.
Let H L ( X ; Y ) denote the collection of
L -Lipschitz functions w : X → Y .
Lemma 2.13.
Suppose that inf M ρ > 0 , and dim ( M ) = m .
Then for any t >
[Equation]
holds with probability at least 1 − 2 t − mm +2 n − ( ct − 1) .
The estimate (10) is called a discrepancy result (Talagrand, 2006; Gy¨orﬁ et al., 2006), and is a uniform version of concentration inequalities.
We include a simple proof in Section A.2.
Proof of Theorem 2.11.
By Lemma 2.13 the event that
[Equation]
for all Lipschitz constants L > 0 has probability one.
For the rest of the proof we restrict ourselves to this event.
Let u n ∈ W 1 , ∞ ( X ; Y ) be a sequence of minimizers of (1), and let u ∗ ∈ W 1 , ∞ ( X ; Y ) be any minimizer of (6).
Then since
[Equation]
we have Lip( u n ) ≤ Lip( u 0 ) =
L for all n .
By the Arzel`a-Ascoli Theorem (Rudin, 1976) there exists a subsequence u n j and a function u ∈ W 1 , ∞ ( X ; Y ) such that u n j → u uniformly as n j → ∞ .
Note we also have Lip( u ) ≤ lim inf j →∞ Lip( u n j ) .
Since
[Equation] [Equation]
it follows from (11)
that L [ u n j , ρ n j ] → L [ u, ρ ] as j → ∞ .
It also follows from (11) that J n [ u ∗ ] →
J [ u ∗ ] as n → ∞ .
Therefore
[Equation]
Therefore, u is a minimizer of J .
By Theorem A.1, u = u ∗ on M , and so u n j → u ∗ uniformly on
M as j → ∞ .
Now, suppose that (9) does not hold.
Then there exists a subsequence u n j and δ > 0 such that
[Equation]
for all j ≥ 1 .
However, we can apply the argument above to extract a further subsequence of u n j that converges uniformly on M to u ∗ , which is a contradiction.
This completes the proof.
Finally, we prove a rate in the case where the loss (cid:96) is strongly convex in the ﬁrst variable.
Deﬁnition 2.14.
We say that (cid:96) is strongly convex with parameter θ > 0 if
[Equation]
for all y 0 , y 1 , y 2 ∈ Y and 0 ≤ t ≤ 1 .
We note that when (cid:96) is twice differentiable, this notion of strong convexity is equivalent to assuming ∇ 2 y 1 (cid:96) ≥ θI .
The deﬁnition in equation (12) is useful for non-smooth functions, such as the Lipschitz semi-norm present in J [ u ] .
Our ﬁnal result is the following L 2 convergence rate in the strongly convex setting.
Theorem 2.15.
Suppose that (cid:96) : Y × Y
→ R is Lipschitz and strongly convex and let L
= Lip( u 0 ) .
Then for any t > 0 , with probability at least 1 − 2 t − mm +2 n − ( ct − 1) all minimizing sequences u n of (1) and all minimizers u ∗ of (6) satisfy
[Equation]
Before proving Theorem 2.15, we require a preliminary lemma.
Lemma 2.16.
If u ∗ ∈ W 1 , ∞ ( X ; Y ) is a minimizer of (6) and u ∈ W 1 , ∞ ( X ; Y ) then
[Equation]
Proof.
We use Proposition A.4 with u 1 = u ∗ and u 2 =
u
to obtain
[Equation]
Since J [ tu ∗ + (1 − t ) u ] ≥ J [ u ∗ ]
[Equation]
and so θ
[Equation]
Setting t = 1 completes the proof.
Proof of Theorem 2.15.
Let L
=
Lip( u 0 ) .
By Lemma 2.
[Equation]
holds with probability at least 1 − 2 t − mm +2 n − ( ct − 1) for any t > 0 .
Let us assume for the rest of the proof that (13) holds.
As in the proof of Theorem 2.11, we have Lip( u n ) ≤ L and Lip( u ∗ ) ≤ L , and so
[Equation]
Therefore
[Equation]
By Lemma
16 we deduce
[Equation]
which completes the proof
A P ROOFS A. P ROOFS FOR CLEAN LABELS
In this section we provide the proof of results stated in § 2.3.
Theorem A.1.
Suppose the loss function satisﬁes Assumption
[Equation]
If u, v ∈ W 1 , ∞ ( X ; Y ) are two minimizers of (6) and inf M ρ > 0 then u
= v on M .
Proof.
Let w
=
( u + v ) / 2 .
Then
[Equation]
Therefore
[Equation]
w is a minimizer of J and so we have equality above
[Equation]
which yields
[Equation]
Since (cid:96) is strictly convex in its ﬁrst argument, it follows that u
= v on M .
Proof of Lemma 2.10 of § 2.3.
There exists (cid:15) M such that for any 0 < (cid:15) ≤ (cid:15) M , we can cover M with N geodesic balls B 1 , B 2 , . . .
, B N of radius (cid:15) , where N ≤ C(cid:15)
− m and C depends only on M (Gy ¨ orﬁ et al., 2006).
Let Z i denote the number of random variables x 1 , . . .
, x n falling in B i .
Then
Z i ∼ B ( n, p i ) , where p i =
(cid:82) B i ρ ( x ) dV ol ( x ) .
Since ρ ≥ θ > 0 and V ol ( B i ) ≥ c(cid:15) m we have p i ≥ c(cid:15) m .
Let A n denote the event that at least one B i is empty (i.e., Z i = 0 for some i ).
Then by the union bound we deduce
[Equation]
Choose 0 < (cid:15) ≤ (cid:15) M in the form n(cid:15) m
= t log( n ) with t ≤ n(cid:15) m M / log( n ) .
Then
P
[Equation]
( ct − 1) log( n )) .
In the event that A n does not occur, then each B i has at least one point, and so | x − σ n ( x ) | ≤ C(cid:15) for all x ∈ M .
Therefore
[Equation]
with probability at least 1 − Ct − 1 exp ( − ( ct − 1) log( n )) .
Since (cid:107) Id − σ n (cid:107) L ∞ ( M ; X ) ≤ C √ d , the result holds for t ≥ n(cid:15) m M / log( n ) , albeit with a larger constant C .
A. P ROOFS FOR NOISY LABELS
Here, we give the proofs of lemmas and propositions required in Section 2.4.
We ﬁrst give the proof of Lemma 2.13.
A key tool in the proof is Bernstein’s inequality (Boucheron et al., 2013), which we recall now for the reader’s convenience.
For X 1 , . . .
, X n i.i.d.
with variance σ 2 = E
[( X i − E
[ X i ]) 2 ] , if | X i | ≤ M almost surely for all
i then Bernstein’s inequality states that for any (cid:15) >
[Equation]
n
[Equation]
Proof of Lemma 2.13.
We note that it is sufﬁcient to prove the result for w ∈ H L ( X ; Y ) with (cid:82)
M wρ dV ol
( x ) = 0 .
In this case, we have w ( x ) = 0 for some x ∈ M , and so (cid:107) w (cid:107) L ∞ ( X ; Y ) ≤ CL .
We also write m in place of m in the proof for simplicity.
We ﬁrst give the proof for M = X
=
[0 , 1] m .
We partition X into hypercubes B 1 , . . .
, B N of side length h > 0 , where N =
h − m .
Let Z j denote the number of x 1 , . . .
, x n falling in B j .
Then
Z j is a Binomial random variable with parameters n and p j = (cid:82) B j ρ dx ≥ ch m .
By the Bernstein inequality we have for each j that
[Equation] [Equation]
provided 0 < (cid:15) ≤ h m .
Therefore, we deduce
[Equation]
holds with probability at least 1 − 2 h − m exp (cid:0) − cnh − m (cid:15) 2 (cid:1) for any 0 < (cid:15) ≤ h m .
Choosing (cid:15) =
h m
+ we have that
[Equation]
holds for all u ∈ H L ( X ; Y ) with probability at least 1 − 2 h − m exp (cid:0) − cnh m +2 (cid:1) , provided h ≤ 1 .
By selecting nh m +2 = t log( n ) sup
[Equation]
L
[Equation]
X ; Y )
[Equation]
holds with probability at least 1 − 2 t − mm +2 n − ( ct − 1) for t ≤ n/ log( n ) .
Since we have (cid:107) w (cid:107) L ∞ ( X ; Y ) ≤ CL , the estimate
[Equation]
trivially holds, and hence we can allow t > n/ log( n ) as well.
We sketch here how to prove the result on the manifold M .
We cover M with k geodesic balls of radius (cid:15) > 0 , denoted B M ( x 1 , (cid:15) ) , . . .
, B M ( x k , (cid:15) ) , and let ϕ 1 , . . . ,
ϕ k be a partition of unity subordinate to this open covering of M .
For (cid:15) > 0 sufﬁciently small, the Riemannian exponential map exp x : B (0 , (cid:15) ) ⊂
T x M → M is a diffeomorphism between the ball B (0 , r ) ⊂
T x M and the geodesic ball B M ( x, (cid:15) )
⊂ M , where T x M ∼ = R m .
Furthermore, the Jacobian of exp x at v ∈ B (0 , r ) ⊂ T x M , denoted by J x ( v ) , satisﬁes (by the Rauch Comparison Theorem)
[Equation]
Therefore, we can run the argument above on the ball B (0 , r ) ⊂ R m in the tangent space, lift the result to the geodesic ball B M ( x i , (cid:15) ) via the Riemannian exponential map exp x , and apply the bound
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
to complete the proof.
Remark A.2 .
The exponent 1 / ( m + 2) is not optimal, but affords a very simple proof.
It is possible to prove a similar result with the optimal exponent 1 /m in dimension m ≥ 3 , but the proof is signiﬁcantly more involved.
We refer the reader to (Talagrand, 2006) for details.
Remark A.3 .
The proof of Theorem 2.11 shows that (1) Γ -converges to (6) almost surely as n → ∞ in the L ∞ ( X ; Y ) topology.
Γ -convergence is a notion of convergence for functionals that ensures minimizers along a sequence of functionals converge to a minimizer of the Γ -limit.
While we do not use the language of Γ -convergence here, the ideas are present in the proof of Theorem 2.11.
We refer to (Braides, 2002) for details on Γ -convergence.
We ﬁnally give a proposition useful in the proof of Lemma 2.16.
Proposition A.4.
If (cid:96) is strongly convex with parameter θ > 0 then
J [ tu 1 + (1 − t ) u 2 ] +
[Equation] [Equation]
(cid:90)
[Equation]
for all u 1 , u 2 ∈ W 1 , ∞ ( X ; Y ) and 0 ≤ t ≤ 1 .
Proof.
We compute
[Equation]
which completes the proof.
B I NDUCED M ATRIX NORMS
In some cases, we can take advantage of explicit formulas for matrix norms, which makes the estimates in (3) an explicit function of the weights.
Deﬁne the induced matrix norm by
[Equation]
Then the following matrix norms formulas hold (see (Horn et al., 1990, Chapter 5.6.4))
[Equation]
C V ARIATIONAL P ROBLEMS IN I MAGE P ROCESSING AND L IPSCHITZ E XTENSIONS
The variational problem (1) can be interpreted as a relaxation of the Lipschitz Extension problem.
[Equation]
subject
[Equation]
for D ⊂ X .
The problem (LE) has more that one solution.
Two classical results giving explicit solutions in one dimension go back to Kirzbaum and to McShane (McShane, 1934).
However solving (LE) is not practical for large scale problems.
There has be extensive work on the Lipschitz Extension problem, see, (Johnson & Lindenstrauss, 1984), for example.
More recently, optimal Lipschitz extensions have been studied, with connections to Partial Differential Equations, see (Aronsson et al., 2004).
We can interpret (1) as a relaxed version of (LE), where λ − 1 is a parameter which replaces the unknown Lagrange multiplier for the constraint.
Variational problems are fundamental tools in mathematical approaches to image processing (Aubert & Kornprobst, 2006) and inverse problems more generally.
Without regularization inverse problems can be ill-posed.
The general form of the problem is
[Equation]
(15) which combines a loss or ﬁdelity functional, L [ u, u 0 ] , which depends on the values of u and the reference image u 0 , and a regularization functional, R [ ∇ u ] , which depends on the gradient, ∇ u .
The parameter λ determines the relative strength of the two terms which emphasize ﬁdelity versus regularization.
Example C.1 .
For example, a typical ﬁdelity term is the standard least-squares L [ u, u 0 ]
= (cid:107)
u − u 0 (cid:107) 2 L 2 ( D ) .
The regularization (cid:107)|∇ u ( x ) |(cid:107) 2 L 2 ( D ) corresponds to the classical Tychonov regularization (Tikhonov & Arsenin, 1977), R [ ∇ u ] = (cid:107)|∇ u ( x ) |(cid:107)
L 1 ( D ) is the Total Variation regularization model of Rudin, Osher and Fatemi (Rudin et al., 1992).
Lipschitz regularization in not nearly as common.
It appears in image processing in (Pock et al., 2010, § 4.4) (Elion & Vese, 2007) and (Guillot & Le Guyader, 2009).
Variational problems of the form (15) can be studied by the direct method in the calculus of variations (Dacorogna, 2007).
The problem (15) can be discretized to obtain a ﬁnite dimensional convex convex optimization problem.
The variational problem can also be studied by ﬁnding the ﬁrst variation, which is a Partial
Differential Equation (Evans, 1998), which can then be solved numerically.
Both approaches are discussed in (Aubert & Kornprobst, 2006).
In Figure 4 we compare different regularization terms, in one dimension.
The difference between the regularizers is more extreme in higher dimensions.