D ISTANCE -B ASED R EGULARISATION OF D EEP N ETWORKS FOR F INE -T UNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We investigate approaches to regularisation during ﬁne-tuning of deep neural networks.
First we provide a neural network generalisation bound based on
Rademacher complexity that uses the distance the weights have moved from their initial values.
This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks.
Our bound is highly relevant for ﬁne-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation.
Inspired by this, we develop a simple yet effective ﬁne-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning.
Empirical evaluation shows that our algorithm works well, corroborating our theoretical results.
It outperforms both state of the art ﬁne-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.
I NTRODUCTION
TheImageNetLargeScaleVisualRecognitionChallengeshaveresultedinanumberofneuralnetwork architectures that obtain high accuracy when trained on large datasets of labelled examples (He et al., 2016;TanandLe,2019;Russakovskyetal.,2015).
Althoughthesemodelshavebeenshowntoachieve excellent performance in these benchmarks, in many real-world scenarios such volumes of data are not available and one must resort to ﬁne-tuning an existing model: taking the weights from a model trained on a large dataset, to initialise the weights for a model that will be trained on a small dataset.
The assumption being that the weights from the pre-trained model provide a better initialisation than randomly generated weights.
Approaches for ﬁne-tuning are typically ad hoc, requiring one to experiment with many problem-dependent tricks, and often a process that will work for one problem will not work for another.
Transforming ﬁne-tuning from an art into a well principled procedure is therefore an attractive prospect.
This paper investigates, from both a theoretical and empirical point of view, the impact of different regularisation strategies when ﬁne-tuning a pre-trained network for a new task.
Existing ﬁne-tuning regularisers focus on augmenting the cross entropy loss with terms that indirectly or directly penalise the distance the ﬁne-tuned weights move from the pre-trained values.
The intuition behindthisseemssensible—theclosertheﬁne-tunedweightsaretothepre-trainedweights, thelessinformation is forgotten about the source dataset—but it is not obvious how this idea should be translated into an effective algorithm.
One should expect that the choice of distance metric is quite important, but existing methods exclusively make use of Euclidean distance (Li et al., 2019; 2018) without any theoretical or empirical justiﬁcation regarding why that metric was chosen.
These methods achieve only a small improvement in performance over standard ﬁne-tuning, and it is reasonable to expect that using a metric more suited to the weight space of neural networks would lead to greater performance.
Moreover, while the use of penalty terms to regularise neural networks is well established, the impact of using penalties vs constraints as regularisers has not been well studied in the context of deep learning.
In order to study the generalisation error of ﬁne-tuned models, we derive new bounds on the empirical
Rademacher complexity of neural networks based on the distance the trained weights move from their initial values.
In contrast to existing theory (e.g., Neyshabur et al.
; Bartlett et al.
Long and Sedghi (2019)), we do not resort to covering numbers or make use of distributions over models to make these arguments.
By deriving two bounds utilising different distance metrics, but proved with the same techniques, we are able to conduct a controlled theoretical comparison of which metric one should use as the basis for a ﬁne-tuning regularisation scheme.
Our ﬁndings show that a metric based on the maximum absolute row sum (MARS) matrix norm is a more suitable measure of distance in the parameter space of convolutional neural networks than Euclidean distance.
Additionally, we challenge the notion that using a penalty term to encourage the ﬁne-tuned weights to lie near the pre-trained values is the best way to restrict the effective hypothesis class.
We demonstrate that the equivalence of penalty methods and constraint methods in the case of linear models (Oneto et al., 2016) does not translate to the context of deep learning.
As a result, using projected stochastic subgradient methods to constrain the distance the weights in each layer can move from the initial settings can lead to improved performance.
Several regularisation methods are proposed, with the aim of both corroborating the theoretical analysis with empirical evidence, and improving the performance of ﬁne-tuned networks.
One of these approaches is a penalty-based method that regularises the distance from initialisation according to the MARS-based distance metric.
The other two techniques make use of efﬁcient projection functions to enforce constraints on the Euclidean and MARS distance between the pre-trained and ﬁne-tuned weights throughout training.
The experimental results demonstrate that projected subgradient methods improve performance over using penalty terms, and that the widely used Euclidean metric is typically not the best choice of metric to measure distances in network parameter space.
R ELATED W ORK
The idea of developing an algorithm to restrict the distance of weights from some unbiased set of reference weights has been explored in various forms to improve the performance of ﬁne-tuned networks.
Li et al.
(2018) presented the (cid:96) 2 -SP regulariser, which consists of adding a term to the objective function that penalises the squared (cid:96) 2 distance of the trained weights from the initial weights.
This is based on an idea initially made use of when performing domain adaptation, where it was applied to linear support vector machines (Yang et al., 2007).
The subsequent work of Li et al.
(2019) follows the intuition that the features produced by the ﬁne-tuned network should not differ too much from the pre-trained features.
They also use Euclidean distance, but to measure distance between feature vectors rather than weights.
The idea is extended to incorporate an attention mechanism that weights the importance of each channel.
The method is implemented by adding a penalty term to the standard objective function.
In contrast to these approaches, we solve a constrained optimisation problem rather than adding a penalty, and we demonstrate that the MARS norm is more effective than the Euclidean norm when measuring distance in weight space.
Many recent meta-learning algorithms also make use of idea that keeping ﬁne-tuned weights close to their initial values is desirable.
However, these approaches typically focus on developing methods for learning the initial weights, rather than working with pre-speciﬁed initial weights.
The model-agnostic meta-learning approach (Finn et al., 2017) does this by simulating few-shot learning tasks during the meta-learning phase in order to ﬁnd a good set of initial weights for a neural network.
Once the learned algorithm is deployed, it adapts to new few-shot learning tasks by ﬁne-tuning the initial weights for a small number of iterations.
Denevi et al.
(2018) proposes a modiﬁed penalty term for ridge regression where, instead of penalising the distance of the parameters from zero, they are regularised towards a bias vector.
This bias vector is learned during the course of solving least squares problems on a collection of related tasks.
Denevi et al.
(2019) extend this approach to a fully online setting and a more general family of linear models.
Previous work investigating the generalisation performance of neural network based on the distance the weights have travelled from their initial values has done so with the aim of explaining why existing methods for training models work well.
Bartlett et al.
(2017) present a bound derived via covering numbers that shows the generalisation performance of fully connected networks is controlled by the distance the trained weights are from the initial weights.
Their bound makes use of a metric that scales with the number of units in the network, which means if the result is extended to a class of simple convolutional networks then the generalisation performance will scale with the resolution of the feature maps.
A similar bound can also be proved through the use of PAC-Bayesian analysis (Neyshabur et al., 2018).
One can make use of different metrics and techniques for applying covering numbers to bounding generalisation that do not have the same implicit dependence on the number of units, but they instead depend directly on the number of weights in the network (Long and
Sedghi, 2019).
Neyshabur et al.
(2019) investigate the performance of two layer neural networks with
ReLU activation functions, demonstrating that as the size of the hidden layer increases, the Frobenius distance (i.e., the Frobenius norm of the difference between initial and trained weight matrices) shrinks.
Inspired by this observation, they show that one can construct a bound on the Rademacher complexity of this class by using Euclidean distance between the initial weights and trained weights of each individual unit.
Although they bound the Rademacher complexity directly, they still incur an explicit dependence on the size of the hidden layer, and their analysis is restricted to fully connected networks with only a single hidden layer.
In contrast to these previous studies, our focus is on designing an algorithm that will improve the performance of ﬁne-tuned networks, rather than explaining the performance of the standard ﬁne-tuning methods that are already widespread.
Therefore, we do not aim to infer what properties of existing methods enable networks to generalise well—we instead derive such properties and then develop algorithms that enforce them.
Moreover, we put a particular emphasis on choosing a metric that is suitable for contemporary convolutional networks, and thus will not scale with the size of the feature maps, while also being easy to implement efﬁciently.
D ISTANCE -B ASED G ENERALISATION B OUNDS
Throughout this section we will analyse a loss class consisting of feed-forward neural networks, f ( (cid:126)x ) =
( φ L ◦ ... ◦ φ 1 )( (cid:126)x ) , where each φ j ( (cid:126)x ) =
ϕ ( W j (cid:126)x ) is a layer with a 1-Lipschitz activation function, ϕ .
Both the norm of the weight matrix of each layer and the distance of the ﬁne-tuned weights from the pre-trained weights are bounded from above.
Formally, we deﬁne
[Equation]
where j goes from 1 to L , l is a ρ -Lipschitz loss function with a range of [0 , 1] , and W
j is the pre-trained weight matrix for layer j .
Each of the pre-trained weight matrices can be either random variables drawn from a distribution with support such that the constraints are always fulﬁlled, or they can be ﬁxed (i.e., drawn from a Dirac delta distribution). The only other requirement is that they are independent of the training data used for ﬁne-tuning. From an empirical perspective it is most useful to consider them non-random quantities.
We have used ∗ as a placeholder for the norm used to measure the magnitude of weight matrices and the distance from the pre-trained weights.
The primary focus is on using the MARS norm,
[Equation]
to prove a bound on the empirical Rademacher complexity (Bartlett and Mendelson, 2002) of F ∞ .
Our main theoretical result, presented below, is obtained by modifying the “peeling”-style arguments typically used to directly prove bounds on the Rademacher complexity of neural network hypothesis classes (for examples, see Neyshabur et al.
; Golowich et al.
Our modiﬁcation of the argument allows us to rephrase the resulting theorem in terms of the distance the parameters can move from their initialisation during training.
Theorem 1.
For all δ ∈ (0 , 1) , the expected loss of all models in F ∞ is, with probability 1 − δ , bounded by
[Equation]
where m is the number of training examples, c is the number of classes, (cid:126)x i ∈ R d , and (cid:107) (cid:126)x i (cid:107) ∞ ≤ C ∞ .
Crucially, when l is chosen carefully (e.g., the ramp loss), the expectation in Theorem 1 is an upper bound for the expected classiﬁcation error rate.
The proof for Theorem 1 can be found in the supplemental material.
The two main terms in this theorem are (i) the product of bounds on the layer norms; and (ii) the summation, which is a bound on the distance the ﬁne-tuned weights can be from the pre-trained weights.
The ﬁrst of these is primarily dependent on the weights obtained via pre-training, whereas (ii) can be controlled during the ﬁne-tuning process.
Moreover, one would expect that if better initial weights are selected via pre-training, then the distance the ﬁnal weights will be from the initial values will be smaller, thus leading to better generalisation.
This is the motivation behind the regularisers we develop in Section 4.
An analogous bound is also derived for the Frobenius norm (i.e., F F ) to facilitate a theoretical comparison between the proposed regularisation method and (cid:96) 2 -SP, an existing approach that relies on penalising Euclidean distance between the pre-trained and ﬁne-tuned parameters.
In order to avoid a direct dependence on the number of parameters (as accomplished in Theorem 1) it is necessary to restrict F F to use only ReLU activation functions.
Theorem 2.
For all δ ∈ (0 , 1) , the expected loss of all models in F F is, with probability 1 − δ , bounded by
[Equation]
where m is the number of training examples, c is the number of classes, (cid:107) (cid:126)x i (cid:107) 2 ≤ C
[Equation]
ϕ ( · ) = ReLU ( · ) , and n j is the number of columns in W j .
We can make several observations by comparing our two bounds with each other and previously published bounds: (i) in contrast to Theorem 1, Theorem 2 incurs a signiﬁcant dependence on the resolution of the intermediate feature maps due to the √ n j factors; (ii) each term in the summations in our bounds can be at most one, whereas the corresponding terms in Bartlett et al.
(2017) can be more than one due to the use of multiple types of norms; (iii) our bound from Theorem 1 does not have a direct dependence on the number of weights, in contrast to the bound provided by Long and
Sedghi (2019).
More detailed comparisons can be found in the supplemental material.
[Equation] T UNING WITH D ISTANCE R EGULARISATION
The analysis presented in Section 3 suggests that the weights in ﬁne-tuned models should be close to the pre-trained weights in order to achieve good generalisation performance.
More speciﬁcally, the learning process should search only within a set of weights within a predeﬁned distance from the pre-trained weights.
We discuss two strategies for accomplishing this: using projected subgradient methods to enforce a hard constraint, and augmenting the standard cross entropy objective with a term that penalises the distance between the pre-trained and ﬁne-tuned weights.
The method based on projection functions is attractive because it guarantees that the constraints will be fulﬁlled even if one uses a heuristic optimisation method to train the network parameters.
However, the penalty-based approaches are more common in the literature, and convenient from an implementation point of view due to the ubiquity of automatic differentiation.
Nevertheless, in contrast to the projection-based methods, the techniques that use penalties have weaker assurances on whether a constraint is actually being enforced.
We discuss the drawbacks of this non-equivalence further in the supplemental material.
4. O PTIMISING WITH P ROJECTIONS
One way to enforce constraints on the weights of neural networks during training is to use a variant of the projected stochastic subgradient method.
This is similar to typical stochastic subgradient methods used when training neural networks, but has the additional step of applying a projection operation after each weight update to ensure that the new weights lie inside the set of feasible parameters that satisfy the constraints.
In the case of classic subgradient descent, in order to guarantee convergence towards a stationary point the projection function must perform a Euclidean projection,
[Equation]
where (cid:99) W are the newly updated parameters that may violate the constraint, W are the projected parameters, and g ( · ) is a convex function specifying the constraint.
Although the Euclidean projection is required for the classic projected subgradient method, other optimisation algorithms may require the projection to be performed with respect to a different metric.
Looking at different optimisers as instantiations of mirror descent with different Bregman divergences is one way to determine the type of projection that should be performed.
Unfortunately some of the most common optimisers used in deep learning, such as Adam (Kingma and Ba, 2015), are not guaranteed to converge even when there are no constraints on the parameters being optimised (Reddi et al., 2018).
This makes extending it to perform constrained optimisation a purely heuristic endeavour, further compounded by the fact that it is also not clear which metric the projection should be performed with respect to.
As such, the projections used in our approaches are performed with respect to the norm that is most convenient from an efﬁciency point of view.
Rather than attempting to constrain the distance between the all pre-trained and ﬁne-tuned weights in the network using a single projection, constraints are applied on a layer-wise basis.
This makes optimisation more manageable and also allows practitioners to favour ﬁne-tuning certain parts of the network—e.g., if one is ﬁne-tuning a network pre-trained on photos to perform a task on paintings where the same underlying classes are present, one might wish to allocate more ﬁne-tuning capacity to earlier layers in the network.
The types of constraints that we wish to enforce take the form
[Equation]
where γ j is a hyperparameter that corresponds to the maximum allowable distance between the pre-trained weights and the ﬁne-tuned weights for layer j .
With a minor rearrangement, this yields a constraint speciﬁcation in the form required for Equation 1,
[Equation]
where we have made the dependence on the pre-trained weights and the hyperparameter explicit.
The resulting optimisation problem is
[Equation]
The remainder of this section presents derivations for the projection functions, π F and π ∞ , corresponding to this constraint speciﬁcation when it is instantiated with the Frobenius norm and the
MARS norm, respectively.
We provide pseudocode in the supplementary material that illustrates how these projections are integrated to the neural network ﬁne-tuning procedure when using a variant of the stochastic subgradient method.
We refer to the Frobenius norm instantiation as (cid:96) 2 -PGM and the MARS norm version as MARS-PGM, where the PGM indicates the use of projection gradient methods.
4.1. C ONSTRAINING F ROBENIUS D ISTANCE
When using the Frobenius distance, Equation 1 can be rewritten as
[Equation]
To simplify the problem, we can instead work on a translated version of the same parameter space where W (0) is the origin.
Setting (cid:98) T = (cid:99)
W − W (0) and T = W − W (0) , the problem becomes
[Equation]
whichistheEuclideanprojectionontothe (cid:96) 2 ballwithradius γ , andhastheknownclosedformsolution
[Equation]
Expanding the deﬁnition of (cid:98) T and translating back into the correct parameter space yields the
Frobenius distance projection function,
[Equation]
4.1. C ONSTRAINING MARS D ISTANCE
The constraint on the MARS distance can be equivalently expressed as a collection of constraints on the (cid:96) 1 distance of each row in the weight matrix from the corresponding row in the pre-trained weight matrix.
That is,
[Equation]
where (cid:126)w i is the i th row of W .
One can then make use of the same translation trick used to derive the
Frobenius distance projection function to change the (cid:96) 1 distance constraints to (cid:96) 1 norm constraints,
[Equation]
where (cid:126)t =
(cid:126)w i − (cid:126)w (0) i .
The problem in Equation 4 is a Euclidean projection onto the (cid:96) 1 ball with radius γ , for which there is no known closed form solution.
There exist algorithms to ﬁnd the (cid:96)projection in time linearly proportional to the dimensionality of the vector (Duchi et al., 2008), but they are not amenable to implementation on graphics processing units due to the sequential nature of the computations involved.
Instead, we apply a projection that minimises the (cid:96) 1 distance between the original point and its projection, subject to the projected point lying inside the (cid:96) 1 ball with radius γ ,
[Equation]
This projection, while not providing the closest feasible point measured in Euclidean distance, still provides a point that satisﬁes the constraints, but is trivial to implement efﬁciently.
Finally, the projection function for the entire weight matrix is given by applying π 1 row-wise, and translating back into the correct parameter space,
[Equation]
where (cid:99) W contains n rows.
4. P ENALTY M ETHODS
One popular approach in the literature to encourage a model to not move too far from a set of initial weights is to augment the loss function with a penalty term.
In our case, this would involve taking the standard objective for the problem at hand (e.g., cross entropy or the hinge loss), and adding penalty terms corresponding to each layer,
[Equation]
where λ j are hyperparameters used to balance the regularisation terms with the main loss function.
Due to the subdifferentiability of the two norms considered in this paper, instantiations of Equationcan be trained via automatic differentiation and a variant of the stochastic subgradient method.
For the
Frobenius norm, we actually penalise the squared Frobenius norm, which recovers the (cid:96) 2 -SP approach of Li et al.
We refer to the instantiation that penalises the MARS distance as MARS-SP.
E XPERIMENTS
This section provides an empirical investigation into the predictive performance of the proposed methods relative to existing approaches for regularising ﬁne-tuning, and also conducts experiments to demonstrate which properties of the novel algorithms are responsible for the change in performance.
Two network architectures are used: ResNet-101 (He et al., 2016), which is representative of a typical large neural network, and EfﬁcientNetB0 (Tan and Le, 2019), a leading architecture intended for use on mobile devices.
Both networks are pre-trained on the 2012 ImageNet Large Scale
Visual Recognition Challenge dataset (Russakovsky et al., 2015).
The Adam optimiser is used for all experiments (Kingma and Ba, 2015).
Information regarding the datasets and hyperparameter optimisation procedure can be found in the supplemental material.
5. P REDICTIVE P ERFORMANCE
The ﬁrst set of experiments are a performance comparison of the proposed methods and existing regularisation approaches for ﬁne-tuning.
The baselines considered are standard ﬁne-tuning with no specialised regularisation, (cid:96) 2 -SP (Li et al., 2018), DELTA (Li et al., 2019), and label smoothing (LS)
[Table]
as formalised in M ¨ uller et al.
Once hyperparameters are obtained, each network architecture and regulariser combination is ﬁne-tuned on both the training and validation folds of each dataset.
The ﬁne-tuning process is repeated ﬁve times with different random seeds to measure the robustness of each method to the composition of minibatches and initialisation of the ﬁnal linear layer, which is trained from scratch architectures in Table 1.
Comparing the average ranks of the methods across different datasets (Dem ˇ sar, 2006), the most salient trend is that the projection-based methods exhibit a signiﬁcant increase in accuracy over their penalty counterparts that use the same distance metrics.
This suggests that, in the case of ﬁne-tuning, using a projection to enforce a constraint on the weights throughout the training process is a better regularisation strategy than adding a term to the objective function that penalises the deviations of weights from their pre-trained values.
Additionally, looking further at the relative performance of the two projection-based regularisers, we can see that the MARS distance variant is more often a better choice than the Frobenius distance.
This observation further supports the conclusions of the comparison of the bounds in Section 3.
5. D ISTANCE FROM I NITIALISATION
To further investigate the relationship between the penalty and projection strategies for regularisation, we analyse the distances between the pre-trained and ﬁne-tuned weights at a per-layer basis.
Figureprovides histograms indicating the distributions of per-layer MARS distances for the models without any regularisation, regularised with MARS-SP, trained with the MARS-PGM, and also DELTA.
All networks were trained on the Pets dataset with the same hyperparameters used by the models examined in Section 5.1.
We observe three trends from these plots.
Firstly, both the penalty and projection methods reduce the MARS distance between the pre-trained and ﬁne-tuned parameters relative to the unregularised model.
Secondly, a signiﬁcant number of the constraints enforced by the projection method are activated—i.e., many of the weight matrices lie on the boundary of the feasible set.
In contrast, the penalty-based regulariser does not enforce similarly activated constraints.
Finally, the results demonstrate that the DELTA method of Li et al.
(2019) does not operate by implicitly regularising the same quantity, as its MARS distance histogram is longer tailed than the others.
5. C APACITY C ONTROL
To demonstrate the ability of the distance-based regularisation methods to control model capacity, we sweep through a range of hyperparameter values and plot the corresponding predictive performance.
Hyperparameters were generated according to λ j = c ˆ λ j and γ j = c ˆ γ j , where c is varied, and ˆ λ j , ˆ γ j are the values found during the hyperparameter optimisation process.
Plots of c versus the resulting accuracy on the pets dataset are given in Figure 2 for both the ResNet101 and EfﬁcientNetB0 architectures.
We can see that the PGM methods behave as the theoretical analysis predicts: hyperparameter [Figure] b e r o f L a y e r
[Figure]
e r o f
L a y e r
[Figure]
[Figure]
[Figure]
e r o f
L a y e r
[Figure]
r
[Figure]
L a [Figure] [Figure] conﬁgurations that lead to very small distances between pre-trained and ﬁne-tuned weights result in underﬁtting, relaxing the hyperparameters too much leads to overﬁtting, and using the optimised hyperparameters (i.e., when c = 1 ) achieves the best performance.
5. E MPIRICAL C OMPARISON OF B OUNDS
We perform an empirical comparison of our two bounds, along with a bound based on the spectral norm (Long and Sedghi, 2019), to demonstrate the relative tightness.
This is done by training neural networks on the MNIST dataset (LeCun et al., 1998).
We use the architecture of Scherer et al.
(2010), which consists of a single convolutional layer with 9 × 9 ﬁlters and 112 channels, followed by 5 ×max pooling, and ﬁnally a linear classiﬁer layer.
The network is trained for 15 epochs using the Adam optimiser (Kingma and Ba, 2015), as training any further does not result in any performance increase.
Following similar work that has performed empirical evaluation of neural network bounds, we evaluate the bounds by computing the relevant norms and distances of the trained network weights.
These quantities are then used in place of the upper bounds (i.e., B ∗ i and D ∗ i ) used in the deﬁnition
1010101010γ
M ea s u r e
FrobeniusMARSSpectral
[Figure]
5 10 15 20 25 30 1010101010γ
M ea s u r e (b)
{+MARS-PGM 0 2 4 6 8 10 
+}
Epoch
M ea s u r e
[Figure]
Epoch
A cc u r ac y
TrainTest
[Figure]
of the hypothesis classes considered by Theorems 1 and 2.
In keeping with previous work that has performed this type of empirical comparison, we measure the distance from random initialisation, rather than ﬁne-tuning from a pre-trained network (Bartlett et al., 2017; Neyshabur et al., 2019).
Figure 3 shows how these three quantities vary for different choices of γ (the same hyperparameter is used for both layers) when using MARS-PGM and (cid:96) 2 -PGM, and also how they differ through the process of training and unregularised model.
We observe several trends: (i) like previous work in this area, all of the empirically evaluated Rademacher complexity bounds are still too loose to be useful for model selection and providing performance guarantees on the expected test set performance; (ii) the bounds based on the MARS norm are consistently tighter (by orders of magnitude) than the bounds based on the Frobenius and spectral norms; (iii) the empirical measurements of model complexity plateau even though the hyperparameters governing the worst-case capacity continue to increase—we suspect this is caused by implicit regularisation from early stopping.
This implicit regularisation from early stopping also provides an explanation for why we observe only a small degradation in performance in Figure 2 when the hyperparameters are set to very large values.
C ONCLUSION
This paper investigates different regularisation methods for ﬁne-tuning deep learning networks.
To facilitate this, we provide two new bounds on the generalisation performance of neural networks based on the distance of the ﬁnal weights from their initial values.
The discussion comparing these bounds suggests that the MARS distance is a more appropriate metric in the parameter space of convolutional networks than Frobenius distance.
Additionally, several new algorithms are presented that enable an experimental comparison between different regularisation strategies.
The empirical results corroborate our theoretical investigation, demonstrating that constraining MARS distance is more effective than constraining Euclidean distance.
Crucially, we also show that, in line with our theoretical results, enforcing a hard constraint throughout the entire training process on the distances the parameters can move is far more effective than the widely used strategy of adding a penalty term to the objective function.
Implementations of the methods used in this paper are available online.
A P ROOF OF T HEOREM
We begin with the deﬁnition of empirical Rademacher complexity.
Deﬁnition 1.
The empirical Rademacher complexity of a class, C , is deﬁned as
[Equation]
where (cid:126)σ is a vector of independent Rademacher distributed random variables and m is the number of training examples.
Using a standard result from statistical learning theory, one can make use of the empirical Rademacher complexity of F ∞ to bound the generalisation error of models contained within the class.
Theorem 3 ((Shalev-Shwartz and Ben-David, 2014, p. 378)) .
For a class, C , containing functions mapping from some set to the interval [ a, b ] , for all δ ∈ (0 , 1) the following holds with probability at least 1 − δ :
[Equation]
where ( (cid:126)x i , y i ) are independent and identically distributed.
We will make use of the following lemma, which uses the “peeling” method of Neyshabur et al.
(2015) to bound the Rademacher complexity of a network based on the product of weight matrix norms.
Lemma 1.
For F ∞ as deﬁned in Section 3, the following inequality holds
[Equation]
i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∞
[Table]
(cid:89)
[Table]
where f k = φ k ◦ ... ◦ φ 1 , (cid:126)x i ∈ R d , and (cid:107) (cid:126)x i (cid:107) ∞ ≤ C ∞ .
Proof.
[Table]
[Equation]
where the ﬁrst inequality is due to the Lipschitz composition property of Rademacher complexities (Ledoux and Talagrand, 1991) and the second follows from H¨older’s inequality.
Iterating this process for all layers of the network results in
[Equation]
[Table]
The expectation can be bounded via Massart’s inequality (Shalev-Shwartz and Ben-David, 2014, p.
383) to conclude the proof.
The following lemma demonstrating how to “split” the Rademacher complexity bound into terms associated with the ﬁne-tuned and pre-trained weights will also be useful.
Lemma 2.
For F ∞ as given in Section 3, the following inequality holds
[Equation]
where f k , (cid:126)x
i
[Equation]
and C ∞ are as deﬁned in Lemma
[Equation]
and (cid:126)v is a ﬁxed vector in R n .
Proof.
[Equation]
where the ﬁrst inequality comes from moving the summation outside the supremum, and the second from multiplying the hypothesis class by a constant and applying the Lipschitz composition property of Rademacher complexities.
This can be further bounded from above by n (cid:88)
[Equation]
where the ﬁrst inequality comes from moving a summation outside the supremum
[Equation]
and the second from H¨older’s inequality.
Because Rademacher complexities are always non-negative
[Equation]
we can omit the absolute value in the deﬁnition of the vector ∞ -norm that one would expect to be enclosing each of the expectations.
The ﬁrst term can be bounded from above by moving the max j inside the expectation and then applying H¨older’s inequality and the deﬁnition of the MARS norm,
[Equation]
Applying Lemma 1 to the resulting expectation completes the proof.
We now prove Theorem 1.
Proof.
The main result of Maurer (2016) tells us that
[Equation]
where ( φ L ◦ ... ◦ φ 1 )
j ( (cid:126)x i ) is the j -th component of the output of ( φ L ◦ ... ◦ φ 1 )( (cid:126)x i ) .
Denoting f k = φ k ◦ ... ◦ φ 1 and the j -th row of W L as (cid:126)w T L,j , the summation over classes can be brought outside the Rademacher complexity and φ L expanded as
[Equation]
to which the Lipschitz contraction inequality can be applied to obtain c
[Equation]
H¨older’s inequality and the deﬁnition of the MARS norm can be used to further bound this as c
[Equation]
of which the ﬁrst expectation can be dealt with via Lemma
[Equation]
yielding
[Equation]
Lemma 2 can be applied to the expectation, yielding
[Equation]
Noting that max k (cid:107) (cid:126)w 0 l,k (cid:107) 1 ≤ B ∞ l , Lemma 2 can be recursively applied to the remaining expectation until the following bound is obtained
[Equation]
The remaining expectation evaluates to zero irrespective of which k is selected by the max, hence nullifying the whole term
[Equation]
and an application of Theorem 3 completes the proof
[Equation]
B P ROOF OF T HEOREM
The proof of Theorem 2 follows the same structure as the proof for Theorem 1.
For conciseness, some details are omitted due to their similarity with the previous proof.
We ﬁrst provide a lemma demonstrating the peeling argument for Frobenius norm-bounded networks.
Lemma 3.
For F F as deﬁned in Section 3, the following inequality holds
[Equation]
where f = φ k ◦ ... ◦ φ 1 , ϕ is the ReLU activation function, and (cid:107) (cid:126)x i (cid:107) 2 ≤ C 2 .
Proof.
Iteratively applying Lemma 1 from Golowich et al.
(2018) results in
[Equation]
and the expectation can be further bounded by
[Equation]
Next, a lemma corresponding to the splitting method is given.
Lemma 4.
For F F as given in Section 3, the following inequality holds
[Equation]
where f k , (cid:126)x i , and C 2 are as deﬁned in Lemma
[Equation]
and (cid:126)v is a ﬁxed vector in R n .
Proof.
From the proof for Lemma
[Equation]
we have
[Equation]
of which the ﬁrst term be further bounded
[Equation]
[Table]
2 B Fj , where the ﬁrst two inequalities are due to Cauchy-Schwarz, and the third is via Lemma 3.
Moving on to the second term, we begin by applying Cauchy-Schwarz, n (cid:88)
[Equation]
which concludes the proof.
Finally, we prove Theorem 2.
Proof.
The proof for this theorem follows the same process as the proof for Theorem 1.
The main result of Maurer (2016) is used to obtain
[Equation]
which, by bringing the summation over c outside the expectation and using the Lipschitz contraction inequality
[Equation]
can be bounded from above by
[Equation]
This can be expanded into two expectations, as in the proof for Theorem
[Equation]
and Lemma 3 applied to the ﬁrst term
[Equation]
yielding
[Equation]
Applying Lemma 4 to the expectation results in
E
[Equation]
As in the proof for Theorem 1, we note that max k (cid:107) w l,k (cid:107) 2 ≤ B Fl and recursively apply Lemmauntil we obtain the following bound
[Equation]
of which the right term is equal to zero.
The result follows from Theorem 3.
C C OMPARISON WITH P REVIOUS B OUNDS
This section provides a comparison of our bounds with previous work that investigates bounding the generalisation gap,
[Equation]
In particular, we consider the main result of Bartlett et al.
[Equation]
and using our notation the main result of Long and Sedghi (2019) is,
[Equation]
These bounds depend on slightly different quantities: we use N to represent the number of parameters in the network, B 2 j and D 2 j indicates upper bounds on the spectral norm and the the associated distance metric, and D 2 , 1 j corresponds to an upper bound on distance metric induced by the matrix (2 , 1) -norm,
[Equation]
In contrast to the aforementioned bounds, which were shown to be true via covering number arguments, the bounds presented in this paper are proved by directly bounding the empirical Rademacher complexity.
As a consequence of these different proof techniques, the constants that appear in each bound are quite different.
The covering number bound both have explicit dependences on the number of parameters in the network—with the bound of Long and Sedghi (2019) being much stronger than that of Bartlett et al.
Our bound does not have this dependence, but it does incur an extra factor of two for every application of the Ledoux-Talagrand contraction inequality used in the peeling argument.
This results in an explicit exponential dependence on the depth of the network.
We note that recent work by Golowich et al.
(2018) shows how the peeling method can be adapted to reduce this to a very mild dependence in some situations.
In addition to inspecting the constants in each bound, one can consider how the norms compare.
The (2 , 1) -norm used in Equation 6, the spectral norm bound from Long and Sedghi (2019), and the
Frobenius norm used in Theorem 2 are all sensitive to the resolution of the intermediate feature maps in convolutional networks, as they do not take into account the parameter sharing in convolutional layers.
In contrast, the MARS norm bound in Theorem 1 is independent of the feature map sizes.
This is because the structure of W in a convolutional layer is a block matrix, where each block is, in turn, a doubly block circulant matrix, so each unique weight in the layer appears at most once in each row.
The MARS norm only considers the “worst” row in the matrix, so each weight is considered at most once, irrespective of feature map resolution.
As such, the MARS bound are more suited to explaining generalisation in convolutional networks.
Furthermore, the way in which the norms are utilised in the bounds are different.
Consider the value of each term in the summations of the bounds; for the bound of Bartlett et al.
(2017), the value of each term can be greater than one, because different norms are used in the numerator and denominator.
Long and Sedghi (2019) do not perform any normalisation, so each term can be greater than one.
In contrast, because the numerator and denominator of the terms in the summations of our bound use the same norms, and from the deﬁnition of our hypothesis class, each term is bounded from above by one.
D A LGORITHM P SEUDOCODE
Algorithm 1 provides high-level pseudocode demonstrating how our proposed projection functions can be integrated with a generic gradient-based optimiser.
Inputs: W (0)1: L , and γ i t ←while W 1: L not converged do t ← t +(cid:99) W ( t ) 1: L ← update ( W ( t − 1) 1: L , ∇ W 1: L f t ( W ( t − 1) 1: L , V ( t − 1) )) for j = 1 to L do
W ( t )
j ← π ( W (0) j , (cid:99) W ( t ) j , γ j ) end for end while
E C ONVERGENCE OF P ENALTY M ETHODS
The usual motivation for using a penalty term to enforce a norm constraint is based on the equivalence of constraints and penalties in the case of linear models with convex loss functions (Oneto et al., 2016).
This equivalence tells us that for each choice of penalty hyperparameter, λ , there is a corresponding setting for a constraint hyperparameter, γ , such that the penalty-based approach and the projectionbased optimiser will search the same hypothesis space.
Although our case is quite different from the setting of Oneto et al.
(2016), since we make use of nonlinear models and apply multiple norm constraints, it is still common to apply penalties to neural networks when one wishes to encourage some sort of structure in the weights of a model.
The technique used to prove the equivalence in the linear case is based on reinterpreting the penalised loss function as the Lagrangian of the constrained problem.
The Karush–Kuhn–Tucker theorem states that stationary points of the Lagrangian are local optimisers of the constrained problem.
The problem with applying this in a deep learning context is that convergence is almost always measured in terms of predictive performance, rather than the training loss.
As a consequence, it is rare that the ﬁne-tuned weights will be a stationary point of the Lagrangian and the constraints implied by the values chosen for the penalty hyperparameter may not be enforced.
In this case, the model resulting from the training process is not guaranteed to come from the pre-speciﬁed hypothesis class.
In contrast, the projection-based methods are guaranteed to provide a model from the chosen hypothesis class even if a stationary point is not found.
We explore this further by examining the when the training loss converges.
Figure 4 shows plots of the training loss, as well as the training and testing accuracy, over the course of training ResNet101 and EfﬁcientNetB0 networks on the pets dataset while applying the MARS-SP regulariser.
From these plots we can see that the training and testing accuracy converges in the expected manner, but the training losseithercontinuestodecreasethroughouttrainingorconvergessigniﬁcantlylaterthanthevalidation accuracy.
This corroborates our hypothesis that users who stop based on train or validation accuracy are likely not to be at or near extrema of the loss, and hence not beneﬁt from any enforced constraints.
F A DDITIONAL E XPERIMENT D ETAILS
We provide a summary of each datasets used in our experiments in Table 2.
We selected datasets from
CVonline 3 that were of a size for which ﬁne-tuning, rather than training from scratch, is required to achieve good performance.
Notably, the CUB-2011 and Stanford Dogs datasets used by prior work (Li et al., 2018; 2019) to evaluate ﬁne-tuning regularisers are not included.
This is because these two datasets overlap with ImageNet, the dataset used during pre-training.
As such, we cannot be sure that the pre-trained networks have not already been trained on the test sets we are using for evaluation.
Like Li et al.
(2018; 2019), we also evaluate on Caltech-256.
However, there are no standard training, validation, and testing splits for this dataset, and the prior work did not release their splits.
As a result, we constructed our own split that consists of 30, 20, and 20 examples per class for the training, validation, and testing folds, respectively.
Epoch
T r a i n i ng L o ss
[Figure]
ResNet-101 Loss       0 .0 .0 .0 .
Epoch [Figure] ac y
TrainTest
[Figure]
{+Accuracy      
+}
Epoch
T r a i n i ng
L o ss
[Figure]
Epoch
[Figure]
TrainTest
[Figure]
All regularisation hyperparameters are tuned using the HyperOpt package developed by Bergstra et al.
(2015), which uses a tree of Parzen estimators to generate promising combinations of hyperparameters based on previously observed losses.
To remain comparable with (cid:96) 2 -SP and DELTA, we make use of only two hyperparameters for each of our proposed methods: the hyperparameter corresponding to the layer trained from scratch ( λ L and γ L for the penalty and constraint methods, respectively), and a single hyperparameter applied to all layers being ﬁne-tuned ( λ j and γ j ).
This also makes the hyperparameter optimisation process reliable, as the number of hyperparameters that must be tuned is not proportional to the number of layers being ﬁne-tuned.
HyperOpt is givensearch iterations and hyperparameter combination with the best validation loss is selected.
F. C OMPARISON USING DELTA E XPERIMENTAL S ETUP
In order to facilitate a direct comparison with previously published results achieved by (cid:96) 2 -SP and
DELTA, we conduct experiments using the same experimental setup as DELTA.
These experiments involve data augmentation; namely, random horizontal ﬂips and random crops.
A known issue with the experimental setup used for evaluation in prior work is the overlap between the ImageNet dataset used for pre-training and the datasets used for evaluation.
We observe two interesting trends in the
[Table] [Table]
results presented in Table 3.
First, our method is still competitive with the existing ﬁne-tuning regularisers when using the ﬂawed experimental protocol utilised by previous work.
Second, there is a very narrow range in performance for all methods compared to the experiments conducted in Section 5.
We suspect this is due to the overlap between the pre-training set and the datasets used for ﬁne-tuning.
Any strategy that keeps the weights or features sufﬁciently close to their pre-trained values will is likely sufﬁcient when there is overlap between the pre-training set and the dataset used for ﬁne-tuning.