A case for new neural networks smoothness constraints
Abstract
How sensitive should machine learning models be to input changes?
We tackle the question of model smoothness and show that it is a useful inductive bias which aids generalization, adversarial robustness, generative modeling and reinforcement learning.
We explore current methods of imposing smoothness constraints and observe they lack the ﬂexibility to adapt to new tasks, they don’t account for data modalities, they interact with losses, architectures and optimization in ways not yet fully understood.
We conclude that new advances in the ﬁeld are hinging on ﬁnding ways to incorporate data , tasks and learning into our deﬁnitions of smoothness
Introduction
How certain should a classiﬁer be when it is presented with out of distribution data?
How much mass should a generative model assign around a datapoint?
How much should an agent’s behavior change when its environment changes slightly?
Answering these questions shows the need to quantify the manner in which the output of a function varies with changes in its input, a quantity we will intuitively call the smoothness of the model.
Today, when learning a smooth function using neural networks the machine learning practitioner is bound to choose between regularization techniques whose effect on smoothness is poorly understood and rigid techniques that do not account for the data or task at hand.
Despite these shortcomings, imposing smoothness constraints on neural networks has led to great progress in machine learning, from boosting generalization and robustness of classiﬁers to increasing the stability and performance of generative models and providing better priors for reinforcement learning agents.
We use the potential of smoothness and the downside of current approaches to construct the case for incorporating tasks, data modality and learning into smoothness deﬁnitions and argue for a more integrated view of smoothness constraints and their interaction with losses, models and optimization.
This paper discusses the smoothness of functions parametrized using neural networks; other classes of functions such as reproducing kernel Hilbert spaces use different notions of smoothness which are outside our scope.
Inside the neural network family of functions, we are looking model smoothness with respect to inputs ; we do not consider smoothness with respect to parameters, but only brieﬂy explore the connection between smoothness with respect to inputs and parameters in Appendix A.1.
Measuring function smoothness
Neural network “smoothness” is a broad, vague, catch all term.
We use it to convey formal deﬁnitions such as differentiable, bounded, Lipschitz, as well as intuitive concepts such as invariant to data dimensions or projections, robust to input perturbations, and others.
One deﬁnition states that a function f : X → Y is n smooth if is n times differentiable with the n -th derivative being continuous.
The differentiability of a function is not a very useful inductive bias, as it is both very local and constructed according to the metric of the space where limits are taken.
What we are looking for is the ability to choose both the distance metric and how local or global our smoothness inductive biases are.
With this in mind, Lipschitz continuity is appealing as it deﬁnes a global property and provides the choice of distances in the domain and co-domain of f .
It is deﬁned as:
[Equation]
where K is denoted as the Lipschitz constant of function f .
Enforcing Equation 1 can be difﬁcult, but according to Rademacher’s theorem if X ⊂ R m is an open set and Y
= R p
then (cid:107) Df ( x ) (cid:107) ≤ K wherever the total derivative Df ( x ) exists.
For p = 1 , this entails (cid:107)∇ x f ( x ) (cid:107) ≤ K wherever f is differentiable.
Conversely, a function that is differentiable everywhere with bounded gradient norm is
Lipschitz.
If f and g are Lipschitz with constants K f and K g , f ◦ g is Lipschitz with constant K f K g .
Since commonly used activation functions are 1-Lipschitz, the task of ensuring a neural network is Lipschitz reduces to constraining the learneable layers to be Lipschitz.
Many neural networks layers are linear operators (linear and convolutional layers, BatchNormalization [1]), and to compute their Lipschitz constant we can use that the Lipschitz constant of a linear operator A under common norms such as l 1 , l 2 , l ∞ is sup
[-=
-][Equation]
(cid:54) =
[Equation]
A x (cid:107) (cid:107) x (cid:107) .
To avoid learning trivially smooth functions and maintain useful variability, it is often beneﬁcial to constrain the function variation both from above and below.
This leads to bi-Lipschitz continuity:
[Equation]
Another way to measure smoothness is through various matrix norms of the Jacobian J ( x ) = df ( x ) d x .
Instead of constraining the total derivative as in Lipschitz continuity, Jacobian metrics account for how each dimension of the function output is allowed to vary as individual input dimensions change.
Smoothness regularization for neural networks
Smoothness regularizers have long been part of the toolkit of the machine learning practitioner: early stopping encourages smoothness by stopping optimization before the model overﬁts the training data; dropout [2] makes the network more robust to small changes in the input by randomly masking hidden activations; max pooling encourages smoothness with respect to local changes; L 2 weight regularization and weight decay [3] discourage large changes in output by not allowing individual weight norms to grow; data augmentation allows us to specify what changes in the input should not result in large changes in the model prediction and thus is also closely related to smoothness and invariance to input transformations.
While common, these smoothness regularization techniques are often introduced as methods which directly target the effects of smoothness - such as generalization and other smoothness beneﬁts discussed in Section 4 - instead of smoothness regularizers.
One explicit approach of targeting smoothness constraints is to regularize the norm of the gradients with respect to inputs of the network (cid:107)∇ x f θ ( x ) (cid:107) , evaluated at different regions of the space [4, 5, 6, 7, 8].
This is often enforced by adding a gradient penalty to the loss function L ( θ ) :
[Equation]
(3) where λ is a regularization coefﬁcient, p reg ( x ) is the distribution at which the regularization is applied, which can either be the data distribution [4, 7] or around it [5, 6], or, in the case of generative models, at linear interpolations between data and model samples [5].
Gradient penalties encourage the function to be smooth around the support of p reg ( x ) either by encouraging Lipschitz continuity ( K (cid:54) = 0 ) or by discouraging drastic changes of the function as the input changes ( K = 0 ).
Beyond gradient penalties, Lipschitz smoothness with respect to the l 2 metric can be encouraged or enforced by individually restricting the maximum eigenvalue, the spectral norm, of each layer of the network.
Spectral regularization [9] uses the sum of the spectrum of each layer as a regularization loss to encourage Lipschitz smoothness.
Spectral Normalization [10] adds a node in the computational graph of the model layers by replacing the weights with their normalized version: L ( W ) becomes
L ( σ ( W )) , where σ ( W ) =
W/
||
W
|| 2 .
Both methods use power iteration to compute the spectral norm of the weight matrix.
Gouk et al.
[11] use a projection method by dividing the weights by the spectral norm after a gradient update.
This is unlike Spectral Normalization, which backpropgates through the normalization operation.
The majority of this line of work has focused on constraints for linear and convolutional layers, and only recently attempts to expand to other layers, such as self attention have been made [12].
Efﬁciency is always a concern and heuristics are often used even for popular layers such as convolutional layers [10] despite more accurate algorithms being available [11, 13].
Parseval networks [14] ensure weight matrices are 1- Lipschitz by enforcing a stronger constraint, orthogonality.
Bartlett et al.
[15] show that any bi-Lipschitz function can be wrriten as a compositions of residual layers [16].
Smoothness for classiﬁcation tasks is deﬁned by Lassance et al.
[17] as preserving features similarities within the same class as we advance through the layers of the network.
The penalty used is (cid:80)
Ll =1 (cid:80)
Cc =1 | σ l ( s c ) − σ l
+1 ( s c ) | , where σ l ( s c ) is the signal of features belonging to class c computed using the Laplacian of layer l .
The Laplacian of a layer is deﬁned by constructing a weighted symmetric adjacency matrix of the graph induced by the pairwise most similar layer features in the dataset.
This type of task dependent approach to smoothness is promising, as we will discuss later.
The beneﬁts of smooth function approximators
Generalization .
Learning models that generalize beyond training data is the goal of machine learning.
The common wisdom is that models with small complexity generalize better [18].
Despite this, we have seen that deep, overparametrized neural networks tend to generalize better [19] and that for
Bayesian methods, Occam’s razor does not apply to the number of parameters used, but to the complexity of the function [20].
A way to reconcile these claims is to incorporate smoothness into deﬁnitions of model complexity and to show that smooth, overparametrized neural networks generalize better than their less smooth counterparts.
Methods that encourage smoothness such as weight decay, dropout and early stopping have been long shown to aid generalization [2, 21, 22, 23, 24].
Data augmentation has been shown to increase robustness to random noise or to modality speciﬁc transformations such as image cropping and rotations [25, 26, 27].
Sokoli ´ c et al.
[4] show that the generalization error of a network with linear, softmax and pooling layers is bounded by the classiﬁcation margin in input space.
Since classiﬁers are trained to increase classiﬁcation margins in output space, smoothing by bounding the spectral norm of the Jacobian increases generalization performance; this leads to empirical gains on standard image classiﬁcation tasks.
Generalization has been recently reexamined under the light of double descent [24, 28], a phenomenon named after the shape of the generalization error plotted against the size of a deep neural network: as the size of the network increases the generalization error decreases (ﬁrst descent), then increases, after which it decreases again (second descent).
We postulate there is a deep connection between double descent and smoothness: in the ﬁrst descent, the generalization error is decreasing as the model is given extra capacity to capture the decision surface; the increase happens when the model has enough capacity to ﬁt the training data, but it cannot do so and retain smoothness; the second descent occurs as the capacity increases and smoothness can be retained.
This view of double descent is supported by empirical evidence which shows that its effect is most pronounced on clean label datasets and when early stopping and other regularization techniques are not used [24].
We later show that smoothness constraints heavily interact with optimization which further suggests that empirical investigations into the impact of smoothness on the observation of double descent are needed.
Reliable uncertainty estimates .
Neural networks trained to minimize classiﬁcation losses provide notoriously unreliable uncertainty estimates; an issue which gets compounded when the networks are faced with out of distribution data.
Recently, Liu et al.
[29] showed that the key component for learning robust, single model neural estimators is combining “distance preserving” feature learning obtained by using bi-Lipschitz neural networks with a Gaussian Process classiﬁer.
Their method is competitive with standard methods of obtaining uncertainty estimates [30] on both vision and language understanding tasks.
These results suggest that having a smooth decision surface is not sufﬁcient to compensate for sharp feature learners and that end to end model smoothness is required.
Robustness to adversarial attacks .
Adversarial robustness has become an active area of research in recent years [31, 32, 33, 34].
Early works have observed that the existence of adversarial examples is related to the magnitude of the gradient of the hidden network activation with respect to its input, and suggested that constraining the Lipschitz constant of individual layers can make networks more robust to attacks [31].
However, initial approaches to combating adversarial attacks focused on data augmentation methods [32, 35, 36, 37], and only more recently smoothness constraints have come into focus [4, 14, 17, 19].
We can see the connection between smoothness and robustness by looking at the desired robustness properties of classiﬁers, which aim to ensure that inputs in the same (cid:15) -ball result in the same function output:
[Equation]
The aim of adversarial defenses and robustness techniques is to have (cid:15) be as large as possible without affecting classiﬁcation accuracy.
Robustness against adversarial examples has been shown to correlate with generalization [38], and with the sensitivity of the network output with respect to the input as measured by the Frobenius norm of the Jacobian of the learned function [4, 19].
Lassance et al.
[17] show that robustness to adversarial examples is enhanced when the function approximator is smooth as deﬁned by the Laplacian smoothness signal discussed in Section 3.
Tsuzuku et al.
[13] show that Equation 4 holds when the l 2 norm is used if (cid:15) is smaller than the ratio of the classiﬁcation margin and the Lipschitz constant of the network times a constant, and thus they increase robustness by ensuring the margin is larger than the Lipschitz constant.
Improved generative modeling performance .
Smoothness constraints through gradient penalties or spectral normalization have become a recipe for obtaining state of the art generative models.
In generative adversarial networks (GANs) [39], smoothness constraints on the discriminator and the generator have played a big part in scaling up training on large, diverse image datasets at high resolution [40] and a combination of smoothness constraints has been shown to be a requirement to get GANs to work on discrete data such as text [41].
The latest variational autoencoders [42, 43] incorporate spectral regularization to boost performance and stability [44].
Explicit likelihood tractable models like normalizing ﬂows [45] beneﬁt from smoothness constraints through powerful invertible layers built using residual connections g ( x ) = x + f ( x ) where f is Lipschitz [46].
More informative critics .
Critics, learned approximators to intractable decision functions, have become a fruitful endeavor in generative modeling, representation learning and reinforcement learning.
By teaching the models they are training which small changes to their output are not task relevant, smooth critics lead to faster convergence and increased performance.
Critics are used in generative modeling to approximate divergences and distances between the learned model and the true unknown data distribution, and have been mainly popularised by GANs.
A critic in a function class F can be used to approximate the KL divergence by minimizing the bound [47, 48]:
[Equation]
While due to the density ratio p ( x ) /q
( x ) in its deﬁnition, the KL divergence provides no learning signal when the model and data distributions do not have overlapping support, choosing F to be a family of smooth functions results in a bound on the KL which provides useful gradients and can be used to train a model [6, 49].
We show an illustrative example in Figure 1a: the true decision surface jumps from zero to inﬁnity, while the approximation provided by the MLP is smooth.
Similarly, training the critic more and making it better at estimating the true decision surface but less smooth can hurt training [50].
It’s not surprising that imposing smoothness constraints on critics has become part of many ﬂavours of GANs [5, 6, 7, 9, 40, 51, 52].
The same conclusions have been reached in unsupervised representation learning, where parametric critics are trained to approximate another intractable quantity, the mutual information, using the Donsker–Varadhan or similar bounds [53, 54].
An extensive study on representation learning techniques based on mutual information showed that tighter bounds do not lead to better representations [55].
Instead, the success of these methods is attributed to the inductive biases of the critics employed to approximate the mutual information.
In reinforcement learning, neural function approximators or “critics” approximate state-value functions or action-state value functions and are then used to train a policy to maximize the expected reward.
Directly learning a neural network parametric estimator of the action value gradients - the gradients of the action value with respect to the action - results in more accurate gradients (Figure 3 in [56]), but also makes gradients smoother.
This provides an essential exploration prior in continuous control, where similar actions likely result in the same reward and observing the same action twice is unlikely due to size of the action space; encouraging the policy network to extrapolate from the closest seen action improves performance over both model free and model based continuous control approaches [56].
Distributional distances .
Including smoothness constraints in the deﬁnition of distributional distances by using optimal transport has seen a uptake in machine learning applications in recent years, from generative modeling [5, 51, 57, 58] to reinforcement learning [59, 60], neural ODEs [61] and fairness [62, 63].
Optimal transport is connected to Lipschitz smoothness as the Wasserstein distance ppf *
MLP approx to f *
[Figure]
ppf
[Figure]
can be computed via the Kantorovich-Rubinstein duality [64]:
[Equation]
The Wasserstein distance is ﬁnding the critic that can separate the two distributions in expectation, but constraints that critic to be Lipschitz in order to avoid pathological solutions.
The importance of the Lipschitz constraint on the critic can be seen in Figure 1b: unlike the KL divergence, the optimal
Wasserstein critic is well deﬁned when the two distributions do not have overlapping support, and does not require an approximation to provide useful learning signal for a generative model.
Consequences of poor smoothness assumptions
Weak models .
Needlessly limiting the capacity of our models by enforcing smoothness constraints is a signiﬁcant danger: a constant function is very smooth, but not very useful.
Beyond trivial examples,
Jacobsen et al.
[65] show that one of the reasons that neural networks are vulnerable to adversarial perturbations is due to invariance to task relevant changes - too much smoothness with respect to the wrong metric.
A neural network can be “too Lipschitz”: methods aimed at increasing robustness to adversarial examples do indeed decrease the Lipschitz constant of a classiﬁer, but once the Lipschitz constant becomes too low, accuracy drops signiﬁcantly [66].
There are two main avenues for being too restrictive in the speciﬁcation of smoothness constraints, depending on where and how smoothness is encouraged.
Smoothness constraints can be imposed on the entire input space or only in certain pockets, often around the data distribution.
Methods which impose constraints on the entire space throw away useful information about the input distribution and restrict the learned function needlessly by forcing it to be smooth in areas of the space where there is no data.
This is especially problematic when the input lies on a small manifold in a large dimensional space, such as in the case of natural images, which are a tiny fraction of the space of all possible images.
Model capacity can also be needlessly restrained by imposing strong constraints on the individual components of the model, often the network layers, instead of allowing the network to allocate capacity as needed.
We can exemplify the importance of where and how constraints are imposed with an example, by contrasting gradient penalties - end to end regularization applied around the training data - and
Spectral Normalization - layerwise regularization applied to the entire space.
Figure 2d shows that using Spectral Normalization to restrict the Lipschitz constant of an MLP to be 1 decreases the capacity of the network and severely affects accuracy compared to the baseline MLP - Figure 2b - or the MLP regularized using gradient penalties - Figure 2c.
Further insight comes from Figure 3, which shows that the gradient penalty only enforces a weak constraint on the model and does not heavily restrict the spectral norms of individual layers; this is in stark contrast with Spectral Normalization which by construction ensures each network layer has spectral norm equal to 1.
To show the effect of data dependent regularization on local smoothness we plot the models’ Lipschitz constants at neighborhoods spanning the entire space in Figure 4.
Each Lipschitz constant is computed using an exhaustive grid search inside each local neighborhood rather than a bound - details are provided in Appendix A.2.
As expected, gradient penalties impose stronger constraints around the training data, while Spectral Normalization has a strong effect on the smoothness around points in the entire space.
This simple example suggests that the search for better smoothness priors needs to investigate
[Figure] [Figure]
The decision surfaces for the same models can be seen in Figure 2.
Smaller means smoother.
where we want functions to be smooth and reexamine how smoothness constraints should account for the compositional aspect of neural networks, otherwise we run the risk of learning trivially smooth functions.
Overlooked interactions with optimization .
We show that viewing smoothness only through the lens of the model is misleading, as smoothness constraints have a strong effect on optimization.
The interaction between smoothness and optimization has been mainly observed when training generative models; enforcing the Lipschitz smoothness of the encoder increased the stability of hierarchical VAEs and led variational inference models to the state of the art of explicit likelihood non autoregressive models [44], while smoothness regularization of the critic (or discriminator) has been established as an indispensable stabilizer of GAN training, independently of the training criteria used [6, 7, 9, 40, 67].
Some smoothness regularization techniques affect optimization by changing the loss function (gradient penalties, spectral regularization) or the optimization regime directly (early stopping, projection methods).
Even if they don’t explicitly change the loss function or optimization regime, smoothness constraints affect the path the model takes to reach convergence.
We use a simple example to show why smoothness regularization interacts with optimization in Figure 5b.
We train an unregularized
MLP classiﬁer on MNIST [68] and observe that the learning rate used affects its smoothness throughout training, without changing testing accuracy.
This shows that imposing smoothness constraints on the same model trained with different learning rates would thus lead to different strengths of regularization and drastically change the trajectory of optimization.
Beyond learning rates, smooth- [Figure] [Figure] ness constraints also interact with momentum.
In the GAN setting, Gulrajani et al.
[5] observed that weight clipping in the Wasserstein GAN critic requires low to no momentum.
Weight clipping has since been abandoned in the favour of other methods, but as we show in Figure 5a and Appendix A. current methods like Spectral Normalization applied to GAN critics trained with low momentum decrease sensitivity to learning rates but perform poorly in conjunction with high momentum, leading to slower convergence and higher hyperparameter sensitivity.
We have shown that smoothness constraints interact with optimization parameters such as learning rates and momentum, and argue that we need to reassess our understanding of smoothness constraints, not only as constraints on the ﬁnal model, but as methods which inﬂuence the optimization path .
Sensitivity to data scaling .
Sensitivity to data scaling of smoothness constraints can make training neural network models sensitive to additional hyperparameters.
Let f ∗ be the optimal decision function for a task obtained from using i.i.d samples from random variable X , and f ∗ c obtained similarly from i.i.d samples obtained from cX .
Since f ∗ and f ∗ c can be highly non linear, the relationship between the smoothness of the two functions is unclear.
This gets further complicated when we consider their closest approximators under a neural family.
The effect of data scaling on the smooth constraints required to ﬁt a model can be exempliﬁed using the two moons dataset: with a Lipschitz constraint of 1 on the model the data is poorly ﬁt - Figure 2d - but a much better ﬁt can be obtained by changing the Lipschitz constant to 10 - Figure 2e - or scaling the data - Figure 2f. Wrong model priors .
The wrong kind of smooth functions can have a similar effect to restricting capacity by introducing the wrong inductive biases. For many regularization techniques it is unclear what kind of smoothness they are encouraging and how strong their effect is on the smoothness of the learned function. Lipschitz smoothness constraints on image models are often speciﬁed with respect to the l 2 norm, which is notorious for not being a meaningful distance metric for natural images. Why are we using the power of feature learning if we are restricting our models to be part of a family of functions which impose constraints that rely on rigid distance metrics?
We have seen that smooth critics can be a catalyst for learning by providing the right signal, but that is only if their similarity measures are relevant for the task.
Paving the path towards smoothness in neural networks
Smoothness regularization of neural networks has brought forward advances in a plethora of machine learning tasks, from supervised to unsupervised and reinforcement learning.
These advances are just scratching the surface of the beneﬁts of smoothness, to explore its full potential we have to use new smoothness deﬁnitions, distance metrics and complexity measures; we need to deﬁne task and modality dependent smoothness constraints applied to learned representations; we have to adopt an integrated view and understand its interactions with losses, data, model architectures and optimization.
Meanwhile, in the world of task agnostic smoothness applied to high dimensional data spaces, we continue to be surprised that specifying smoothness constraints is not better.
New ways of deﬁning smoothness .
Improving model generalization and robustness requires specifying the right level of invariance by using task information to deﬁne smoothness constraints.
To solve issues in generative modelling such as “mode dropping” or “mode collapse”, where entire modes from the data distribution are not captured by the model, we have to go beyond current smoothness measures such as Lipschitz continuity deﬁned on the input space.
With the right feature space, images of cats are close to images of dogs, and thus a model which is smooth in that space is less likely to drop one of the two modes.
We have to ask what are the desired properties of h such that (cid:107) f
( h ( x ))
− f ( h ( y ))
(cid:107) ≤ (cid:107) h ( x ) − h ( y ) (cid:107) instead of applying the smoothness constraints on the raw data.
Since we require that the mapping h does not discard task relevant information in the data, maintains useful diversity and accounts for input modalities, it has to be data and task dependent.
As we have seen again and again in the development of machine learning, handcrafting h is not a scalable solution, and thus the mapping itself has to be learned .
Since the approaches used for learning the right representations have to be task dependent, different insights will be required for supervised and unsupervised methods.
We expect that semi-supervised learning will play an important role, and that hints for useful properties of representation domains will come from representation learning methods, data augmentation and inference techniques.
New ways of measuring smoothness .
Measuring smoothness of a function parametrized by a neural network is challenging even for the most common measure of smoothness used in machine learning, Lipschitzness.
Loose upper and lower bounds which rely on function composition are often used [69, 70].
Fazlyab et al.
[66] provide an algorithm with tighter bounds by leveraging that activation functions are derivatives of convex functions and cast ﬁnding the Lipschitz constant as the result of a convex optimization problem.
However, their most accurate approach scales quadratically with the number of neurons and only applies to feed forward networks.
Sokoli´c et al.
[4] provide an upper bound for the Lipschitz constant of a neural network with linear, softmax and pooling layers restricted to input space X via (cid:107) f
( x ) − f ( y ) (cid:107) 2 ≤ sup
z ∈ convex_hull
( X ) J ( z ) (cid:107) x − y (cid:107) 2 , ∀ x , y ∈
X , but empirically resort to layerwise bounds.
If we want to understand the effects of network architectures, regularization methods and optimization algorithms have on model smoothness, we have to be able to accurately measure it.
New learning paradigms .
Combining non parametric methods with feature learning is a promising approach to learning smooth decision surfaces.
Given the right data representations and the appropriate distance metrics, interpolating between training examples is an excellent and interpretable smoothness prior.
Pursuing this avenue of research entails learning the right features, which themselves might have to be smooth [29], as well as further avenues for scaling non parametric methods such as Gaussian Processes, Support Vector Machines and Nearest Neighbours methods to large datasets.
New measures of model complexity .
Standard complexity measures, from VC dimensions and
Rademacher complexity, to simpler measures such as number of learned parameters ignore the problem the model has to solve.
Task deﬁnitions need to be accounted for in the new generation of model complexity measures, since ﬁtting random labels (as per Rademacher complexity) discounts the inductive bias in smoothness constraints that can help model ﬁtting and generalization.
The issue of measuring model complexity is inherently tied with many other issues discussed so far, such as choosing ways to deﬁne and quantify smoothness.
New approaches to old problems .
Smooth learned critics have advanced the state of art in generative modeling and reinforcement learning.
Why stop there?
By viewing parametric critics as learned loss functions, and observing that for any lower layer of a neural network the upper layers are part of a learned loss function, we can further explore and expand the beneﬁts of smoothness.
By exploring how smoothness helps critics in non stationary environments such as reinforcement learning and generative models, we can solve notorious neural network training problems such as covariate shift.
A Appendix A.1 Connection to parameter smoothness
While in we focused on smoothness to variations in the input, it’s worth noting the importance of smoothness to variations of parameters, which has been shown to help optimization and aid generalization.
Speciﬁcally, there are works showing the importance of the Lipschitz constant of the loss function with respect to changes in parameters.
Perhaps the simplest and most common is to add gradient penalties wrt to parameters to the loss function BatchNormalization [1], one of the most popular techniques in deep learning, was originally motivated as a minimizer of covariance shift, but has later been shown not to reduce covariance shift but make both the loss function and its gradients more Lipschitz, both empirically and theoretically [71].
This in turn increases training stability, as well as convergence speed by allowing for higher learning rates.
Activation normalization - normalizing activations by their l p norm and then shifting them by a chosen mean - was proposed by Santurkar et al.
[71] in order to further explore the effects of BatchNormalization.
Activation normalization was shown to have the same performance as BatchNormalization, and its effects are also attributed due to a smoothing of the loss landscape.
Weight standardization [72] normalizes convolutional weight matrices to have zero mean and unit variance across the output channel dimension.
If if another normalization layer follows weight standardization (such as group normalization), weight standardization decreases the norm of the loss gradients with respect to parameters and thus decreases the Lipschitz constant of the loss.
The authors show that the gradients of the loss with respect to parameters are now more Lipschitz.
Another reason to look for smoothness with respect to parameters is gradient variance.
The variance of gradients obtained using the "reparametrization trick" [42, 43] is bounded by the Lipschitz constant of the cost function [73, 74].
The reparametrization trick is a distributional reparametrization technique used in generative modeling - GANs, VAEs - but also in continuous control - SVG [75].
In all of the above, the loss function for the reparametrized components contains another learned module (for
VAEs, it is the decoder, for GANs case, it is the generator, for SVG, it is the action value critic).
Thus, making these learned components Lipschitz should stabilize training.
In the GAN case, regularizing the discriminator has been observed to stabilize training [9] but the motivation provided were varied, from convergence guarantees [13] to requirements of the Wasserstein distance [51].
A way to see the connection between smoothness with respect to parameters and smoothness with respect to inputs discussed in the main paper is to look at backpropagation for a linear layer with weights W ( l ) , where h l +
=
W h l , which resides inside a network trained with loss function L and output layer o :
[Equation]
Equation 7 shows the connection between the gradients used for learning,
[Equation]
and the gradients of the output layer of the network with respect to intermediate activations d
[Equation]
+
[Equation]
A.2 Additional experimental results and experimental methodology
Architectures .
In all the two moons experiments, the DeepMLP has 4 layers and 100, 100, 100 andoutput units respectively.
The shallow MLP has 2 layers of 100 and 1 unit.
All methods were trained for 100 iterations on 50 datapoints.
The MNIST plots are obtained from MLP classiﬁers havinglayers of 1000, 1000, 1000 and 10 units each and are trained for 500 iterations at batch size 100, reaching an accuracy of 95% on the entire test set.
For the GAN CIFAR-10 experiments, we use the architectures speciﬁed in the Spectral normalization paper [10].
Unless otherwise speciﬁed, we use the default Adam optimizer [76] β 1 and β 2 parameters.
Computing the local Lipschitz constant in Figure 4 .
To compute the local Lipschitz function of the decision surface learned on two moons, we split the space into small neighborhoods ( equally sized grids).
For each grid, we sample 2500 random pairs of points in the grid and report max (cid:107) f ( x ) − f ( y ) (cid:107) / (cid:107) x − y (cid:107) .
Spectral normalization .
In Figure 6 we show that the effect of momentum on spectral normalization is independent of whether caching of the initialization vector for power iteration is performed or not.
[Figure]