G RADIENT - BASED OPTIMIZATION IS PROBABLY NOT NECESSARY FOR GENERALIZATION IN NEURAL NETWORKS
A BSTRACT
It is commonly believed that the implicit regularization of optimizers is needed for neural networks to generalize in the overparameterized regime.
In this paper, we observe experimentally that this implicit regularization behavior is generic , i e. it does not depend strongly on the choice of optimizer.
We demonstrate this by training neural networks using several gradient-free optimizers, which do not benefit from properties that are often attributed to gradient-based optimizers.
This includes a guess-and-check optimizer that generates uniformly random parameter vectors until finding one that happens to achieve perfect train accuracy and a zeroth-order Pattern Search optimizer that uses no gradient computations.
In the low sample and few-shot regimes, where zeroth order optimizers are most tractable, we find that these non-gradient optimizers achieve test accuracy comparable to SGD.
The code to reproduce results can be found at https://github.com/Ping-C/optimizer
I NTRODUCTION
The impressive generalization of deep neural networks continues to defy conventional wisdom, where overparameterization relative to the number of data points is thought to hurt model performance.
From the perspective of classical learning theory, using measures such as Rademacher complexity and VC dimension, as we increase the complexity of a model class, the generalization performance of learned models is more likely to deteriorate.
However, in the case of deep learning models, we observe the exact opposite phenomenon – as we increase the number of model parameters, the performance continues to improve.
This is particularly surprising since deep neural networks were shown to easily fit random labels in the overparameterized regime (Zhang et al., 2017).
This combination of empirical and theoretical pointers lead to a large gap in our understanding of deep learning, which has sparked significant interest into studying various forms of implicit bias that could explain generalization phenomena.
Perhaps the most widely-held hypothesis posits that gradient-based optimization gives rise to implicit bias in the final learned network leading to better generalization (Arora et al., 2019; Advani et al., 2020;
Liu et al., 2020; Galanti & Poggio, 2022).
For example, (Arora et al., 2019) showed that deep matrix factorization, which can be viewed as a highly simplified neural network, is biased towards solutions with low rank when trained with gradient flow.
Indeed, (Galanti & Poggio, 2022) shows theoretically and empirically that stochastic gradient descent (SGD) with a small batch size can implicitly bias neural networks towards matrices of low rank.
A related concept was used by (Liu et al., 2020) to show that gradient agreement between examples is indicative of generalization in the learned model.
In this paper, we empirically examine the hypothesis that gradient dynamics is a necessary source of implicit bias for neural networks.
Our investigation is based on a comparison of several zeroth order optimizers, which require no gradient computations, with the performance of SGD.
We focus our studies on the small sample regime where zeroth order optimizations are tractable.
Interestingly, we find that all the gradient-free optimizers we try generalize well compared to SGD in a variety of settings including MNIST (LeCun et al., 2010), CIFAR-10 (Krizhevsky, 2009), and few-shot problems (Bertinetto et al., 2019; Vinyals et al., 2016).
Even though we use fewer samples in our experiments compared to standard settings, this low-data regime highlights the role of model bias, where the generalization behavior of neural networks is particularly intriguing.
The model we test has more than 10 , 000 parameters, but it has to generalize with fewer than 1 , 000 training samples.
Without implicit bias, such a feat is nearly impossible in realistic use cases like the ones we consider.
Our work shows empirically that generalization does not require the implicit regularization of gradient dynamics, at least in the low-data regime.
It is still an open question whether gradient dynamics play a larger role in other regimes, e.g., where more data is available.
We need to caution that we are not claiming that gradient dynamics have no effect on generalization, as it has been clearly shown both theoretically and empirically that it has a regularizing effect (Arora et al., 2019; Galanti & Poggio, 2022).
Instead, we argue that the implicit regularization of gradient dynamics is only secondary to the observed generalization performance of neural networks, at least in the low-data regimes we study.
The observations in this paper supports the volume hypothesis for generalization: The implicit bias of neural network may arise from the volume disparity of different basins in the loss landscape, with good hypothesis classes occupying larger volumes.
The conjecture is empirically supported by the observation that even a “guess & check algorithm,” which randomly samples solutions from parameter space until one is found with low training error, can generalize well.
The success of this optimizer strongly suggests that generalizing minima occupy a much larger volume than poorly generalizing minima in neural loss functions, and that this volume disparity alone is enough to explain generalization in the low-shot regime.
Finally, we show in a previously studied toy example that volume implicitly bias the learned function towards good minima, regardless of the choice of optimizer.
R ELATED W ORK
The capability of highly overparametrized neural networks to generalize remains a puzzling topic of theoretical investigations.
Despite their high model complexity and lack of strong regularization, neural networks do not overfit to badly generalizing solutions.
From a classical perspective, this is surprising.
Bad global solutions do exist (Zhang et al., 2017; Huang et al., 2020), yet usual training routines that optimize neural networks with stochastic gradient descent never find such worst-case solutions.
This has led a flurry of work re-characterizing and investigating the source of the generalization ability of neural networks.
In the following we highlight a few angles.
High-Dimensional Optimization Before reviewing the literature on gradient dynamics, we want to review the underlying reasons why gradient-based (first-order) optimization is so central to deep neural networks:
The core reasons for this is often dubbed the curse of dimensionality : For arbitrary optimization problems (with minimal conditions, i.e. (Noll, 2014)) a first-order optimizer will converge to a local minimal solution in polynomial time in the worst-case, independent of the dimensionality of the problem.
However, a zeroth order algorithm without gradient information will have to, in the worst-case, evaluate a number of queries that increases exponentially with the dimensionality of the problem, even for smooth, convex optimization problems (Nesterov, 2004).
However as we will discuss, neural networks are far from a worst-case scenario, given that many solutions exist due to the inter-connectedness of minimas in neural networks.
Gradient Dynamics
Here we briefly review literature that argues gradient descent as the main implicit bias for generalization of neural networks.
In Liu et al.
, they argue that deep networks generalize well because of the large agreement of gradients among training examples using a quantity called gradient signal to noise ratio (GSNR).
They found both empirically and theoretically that a large GSNR would lead to better generalization and that deep networks induce a large GSNR during training, leading to better generalization.
Arora et al.
(2019) show that the dynamics of gradient-based optimization induce implicit bias that is stronger than typical norm-based bias in the setting of deep matrix factorization, and raise the question whether implicit biases can be induced from first-order optimization that cannot be captured by any explicit regularization.
Advani et al.
(2020) argues that in the overparameterized regime, the gradient dynamics prevent learning from happening in certain subset of weights, which effectively works as implicit regularization.
A recent paper by (Galanti & Poggio, 2022) proves that SGD trained networks have a low rank implicit bias and hypothesizes that such an implicit bias may be the source of superior generalization for deep neural networks.
Non-gradient Based Explanation of Implicit Bias Several works have tried to explain the generalization behavior of neural networks with non-gradient based implicit regularization.
Neyshabur et al.
(2015) argues weight norms to be the main measure of capacity control that allows neural networks to generalize.
Keskar et al.
(2016) suggests that flatness in the parameter space corresponds to simpler functions, thus allowing neural networks to generalize.
However, Dinh et al.
(2017) later show that when the flatness measure is not scale-invariant, sharp solutions can generalize just as well with appropriate rescaling of the network parameters.
Valle-Perez et al.
(2018) argue that the parameter-function map is exponentially biased towards simple functions.
Rahaman et al.
(2019) shows that neural networks are biased toward low frequency functions that vary globally without local fluctuation.
Among all works that try to explain neural network generalization, most recent works argue gradient descent or stochastic gradient descent as the main implicit bias of neural network training that allows deep overparameterized networks to generalize.
Volume and Bayesian Modeling From a Bayesian perspective, flat minima of the loss surface are highly represented in the Bayesian model average, especially when they contain functional diversity (Wilson & Izmailov, 2020).
The size of a posterior peak has also been connected to Occam factors indicating that they represent simpler solutions which are more compressible and generalize well (MacKay et al., 2003).
Smith & Le (2017) studies generalization behavior of overparameterized linear models where they find that the Bayesian evidence or marginal likelihood, which is connected to generalization, strongly favors flat minima.
A line of work on PAC-Bayes generalization bounds, which is related to compressibility and the Bayesian evidence, uses compressibility to guarantee generalization and finds the flat minima are more compressible as they yield more bits back from the KL-divergence term in the bound (Dziugaite & Roy, 2017).
In contrast to these works, our findings focus not on why flat minima generalize well but rather demonstrate that we can find them with or without gradient-based optimization.
Similar Lines of Inquiry Mingard et al.
(2021) empirically show that when sampling from wide networks conditioned on the training set, the sampled models behave similarly to finite width networks trained with SGD.
They approximate the posterior with Neural Network Gaussian Process, which is not exact in finite width networks.
Geiping et al.
(2022) show that full batch gradient descent, when coupled with explicit regularization can perform comparably to model trained with
SGD, thus bringing into question the importance of SGD for generalization.
Similar in spirit, we argue that SGD and all gradient-based optimizers are not the main source of generalization behavior of neural networks.
T HE M YSTERY OF G ENERALIZATION WITH O VERPARAMETERIZATION
In this section, we illustrate how the complexity of a hypothesis class increases with the number of parameters in the context of a simple classification problem.
Specifically, we increase the number of hidden units of a two layer neural network and showcase the increasing complexity of the model class.
Despite this increased complexity, SGD often consistently finds good classifiers.
Then, we proceed to show that a similar generalization behavior can be achieved without any gradient dynamics.
We begin with a toy classification problem defined over two classes where the data distribution is a wedge “ Λ ” with a vertical margin separating the two classes (see Figure 1).
Throughout this section, our training and testing data consist of 11 points and 5 points, respectively, each sampled uniformly at random. [Figure]
3. O VERPARAMETERIZATION INCREASES MODEL COMPLEXITY
To illustrate how the model complexity increases with number of parameters, we first poison a model by minimizing the loss on the training data while maximizing the loss on the testing data.
Given we only examine cases where the model (trained by SGD) achieves 100% training accuracy, this represents the worst-case decision boundary for the unpoisoned loss.
As we increase the number of hidden units from 2 to 20, the decision boundary becomes much more ill-behaved (see Figure 2a).
When the hypothesis class is restricted to 2 hidden units (the left most plots in Figure 2a), the model can only fit the data by using a single kink in the decision boundary, so it has to trade off either fitting the training examples to 100% accuracy or performing badly on the poison objective.
Given that the model fits training data, it has to perform well on testing data.
This is consistent with the under-parameterized regime and with classical learning theory.
As we increase the model size, the model class now contains strange decision boundaries that can fit training data while performing poorly on the testing data.
From the perspective of classical learning theory, we would expect models with 20 hidden units to perform much worse than the model with 2 hidden units.
Surprisingly, even though more complicated decision boundaries are available as we increase the model size, we never see such boundaries when optimizing the (unpoisoned) training objective with SGD.
For example, in Figure 2b, as we increase the number of hidden units, the decision boundary remains relatively consistent.
Given that both the weird and nice decision boundaries exist in the model class, it is natural to ask what biases the learned network towards good vs. bad optima.
Due to the consistent behavior of SGD trained networks in the overparameterized regime, it is only reasonable that people started investigating gradient dynamics as a source of implicit regularization (Galanti & Poggio, 2022; Arora et al., 2019; Advani et al., 2020; Liu et al., 2020).
However, we show below that, rather surprisingly, we can obtain similar generalization behavior on the toy problem by using a Guess & Check algorithm that is completely free of gradient dynamics.
3. G ENERALIZING ON TOY PROBLEM WITHOUT GRADIENTS
In our toy setting, we find that generalization is surprisingly generic with respect to the dynamics of optimizers.
To avoid using optimizers with the same inductive biases as gradient methods, we experiment with Guess & Check: we repeatedly sample parameters until a model achieves 100% training accuracy with train loss below a certain threshold.
Unlike other optimizers, we do not use any gradient information, and we also do not take any iterative steps.
Surprisingly, even with Guess & Check, we often end up with a well-behaving decision boundary like the one that we trained with regular SGD see Figure 2c.
From the simple two class toy problem, we can see clearly that Guess & Check solutions already endow the learned model with a very strong implicit bias that does not originate from gradient dynamics.
In the next section, we extend a similar analysis to common datasets such as MNIST &
CIFAR10 to see whether this observation continues to hold in more practical settings.
E XPERIMENTS 4. N ON - GRADIENT B ASED O PTIMIZERS
In our experiments, we test three different non-gradient based optimizers: Guess & Check, Pattern
Search, and Greedy Random Search on varying scales of MNIST & CIFAR-10 and on different architectures.
Here, we explain each optimizer.
4.1. G UESS & C HECK
Under the Guess & Check algorithm, we randomly draw a parameter vector uniformly from [ − 1 , 1] and independently for each parameter.
If the randomly sampled model achieves 100% training accuracy and achieves training loss below a certain threshold, then the model is kept and the optimizer terminates.
With this optimizer, the likelihood that a set of solutions are selected is exactly proportional to the volume of solutions in parameter space.
If a model generalizes well when trained with Guess & Check, then we know that solutions which generalize well have large volume among low-loss parameters.
We do want to make a distinction between flat solutions (Keskar et al., 2016) and solutions with large volumes.
It is possible that a collection of solutions has a very large volume, but is not itself a flat basin by containing many small volume regions that have large volume in aggregate.
When we train with Guess & Check, we can also be confident that gradient based implicit regularization plays no role in the final performance.
In Appendix C, we also show that the results do no differ materially when using a different sampling interval.
Na¨ıve guess-and-check suffers from the problem that the cost of interpolating the training data becomes exponentially expensive as the number of training examples or classes increase, so we have restricted experiments with Guess & Check to smaller sample sizes.
4.1. L OCAL N ON - GRADIENT B ASED O PTIMIZER
Due to the difficulty of scaling Guess & Check algorithm to large problems, we explore two alternative non-gradient based optimizers, Pattern Search and Random Greedy Search, that work for bigger datasets.
Like SGD, both approaches update the model using a local search and may have biases that originate from factors other than volume alone.
The success of Pattern Search and Random Greedy demonstrates that gradients are not strictly needed to observe implicit regularization, but we caution that they may exploit regularization properties of local search that are also exploited by SGD.
Pattern Search Pattern Search randomly selects a parameter in the model and takes a step of fixed size that is randomly chosen to be either positive or negative, if the model achieves lower loss after taking the step, then the parameter is accepted as the new starting point.
If Pattern Search fails to find a step that decreases the loss after going through all the parameters, then the step size is decreased [Table] by a constant factor.
We repeat this procedure until a solution is found that achieves 100% training accuracy.
In our experiments, we use a starting radius of 1, and we decrease the radius by a factor of 2 when it fails to find a descent direction.
Random Greedy Search Random Greedy Search adds Gaussian noise to the initial parameter vector with standard deviation of σ .
If the noised solution improves training loss, then the noised solution is accepted as a new starting point.
If no solution is found after a fixed number of steps, then σ is decreased by a chosen factor before the search continues.
Again, we repeat this procedure until a parameter is found that achieves 100% training accuracy.
In our experiment, we start the procedure with σ = 1 .
If we fail to find a perturbation that decreases loss after 30000 random steps, then we decrease σ by a factor of 2.
4. R ESULTS ON 2-C LASS CIFAR-10/MNIST
[Table]
In this section, we apply the Guess & Check algorithm on a conventional LeNet model on MNIST (LeCun et al., 2010) and CIFAR-10 (Krizhevsky, 2009).
Due to the exponential time complexity of the Guess & Check algorithm, we stick with 2-class problems with fewer thantotal training samples.
To enable fair comparisons between G&C and SGD optimized models, we compare the performance of the models across different train loss levels after the the model’s weights have been normalized.
This is crucial for a fair comparison because it has been observed that lower loss level corresponds to better generalization even after train accuracy has reached 100%.
We find that given the same loss level and number of samples, Guess & Check performs comparably to SGD, especially at lower loss levels.
In the case of CIFAR-10, Guess & Check even outperforms SGD solutions by a substantial margin.
This result is made even more interesting given that the models are capable of pathological overfitting: they are able to completely misclassify the validation set while achieving 100% training accuracy (see Table 1).
To illustrate how well G&C performs, we also train a linear model on MNIST for comparison.
Even though the hypothesis class is now restricted to only linear solutions, a significantly smaller hypothesis class compared to LeNet, the linear model still underperforms the Guess & Check solution on
LeNet by more than 10% in many cases.
Note that, despite being convex, solutions of the linear problem vary because we use SGD and apply early stopping when the desired loss level is achieved.
Even though the number of samples is small, we do note that this regime highlights the effects of overparametrization.
For example, in our LeNet for MNIST, we have 11074 parameters, which is [Table] [Table] orders of magnitude larger than the number of examples.
Yet the model continues to generalize well relative to SGD, showing us that the large volume of the good solution set is on its own enough to bias the optimizer towards favorable generalization.
Even though the generalization performance is similar between the SGD and G&C solutions, we do note that the test accuracies are not exactly the same between models trained with both methods, implying that SGD may have additional bias that G&C does not take into account.
However, our main argument is that optimizer-specific bias is not needed to explain generalization, and may not even be the primary cause of generalization behavior; in our experiments here, the bulk of generalization can be explained by the geometry of the loss landscape.
4. R ESULTS ON 10- CLASS CIFAR-10/MNIST
In this section, we evaluate the importance of gradient based optimizers in the setting where more classes are involved.
However, the Guess & Check algorithm is no longer feasible due to the exponential time complexity.
Instead of Guess & Check, we employ Pattern Search and Greedy Random
Search to evaluate the dependence of generalization on gradient based optimizers.
Again, we find that these non-gradient based optimizers offer similar levels of generalization benefits as SGD despite not using any gradient information at all.
In Table 4, we see that Greedy Random Search and Pattern Search both generalize comparably to SGD.
The average performance difference is only 0.9% across different sample sizes, datasets, and optimizer combinations.
In several cases, Pattern Search even performs better than SGD.
Even though 0.9% may seem large when viewed from the perspective of achieving state-of-the-art accuracy, we note that a performance difference of 0.9% is within the margin that can be expected from hyperparameter tuning, and that we have not tuned either of the zeroth order optimizers, and yet they still achieve a comparable level of generalization to SGD. [Table]
4. F EW -S HOT L EARNING WITH R ES N ETS
In this section, we evaluate the importance of gradient-based optimizers in the few-shot setting where we can evaluate the importance of gradient-based optimization on an industrial-scale model.
For the most part, we find that non-gradient based optimizer Pattern Search performs comparably to
SGD in the 1-shot setting.
Few-shot learning is usually used to test the ability of models to generalize to unseen tasks given limited training examples.
This is a perfect evaluation task for our hypothesis for the following reasons: First, in few-shot learning, we only use 1 or 5 training images per class during the evaluation stage which makes zeroth order optimization possible.
Second, few-shot learners usually utilize a pre-trained feature extractor on the base classes, and only learn a new classification head by SGD or other solvers such as SVM and ridge regression (Lee et al., 2019; Bertinetto et al., 2019) given the unseen tasks.
This setting limits the dimension of learnable parameters, thus making training deeper networks such as ResNet possible with these non-gradient based optimizers. Finally, although we only attempt to learn a single layer, due to the few training examples (1 or 5 per class), we will still have an overparameterized model which is the setting we are interested in. over 600 episodes and the one standard error. Compared to SGD, Pattern Search can always achieve better performance by a large margin.
[Table]
We evaluate the effectiveness of non-gradient based optimizers on CIFAR-FS (Bertinetto et al.
, 2019)and mini-ImageNet (Vinyals et al., 2016) with ResNet-12, a commonly used architecture in the few-shot classification literature.
During the training stage, we pre-train a feature extractor on the base classes and evaluate the generalization on unseen tasks via 1-shot5-way episodes, where each episode is a 5-way classification problem and each class contains 1 training image.
During the evaluation stage, given the unseen episodes and pre-trained feature extractor, we learn a new classification head with a specific optimizer and evaluate the performance on the testing images.
We compare the testing accuracy on 600 different episodes between SGD, and Pattern Search in Table 5.
For Pattern Search, instead of stopping optimization immediately after fitting all the training examples, we keep updating the model until t steps, where we set t = 3000 for both CIFAR-FS and mini-ImageNet.
As showed in Table 5, Pattern Search always outperforms SGD by a large margin, i.e., over 2% for both CIFAR-FS and mini-ImageNet, which suggests that gradient-based optimizers are not necessary for generalization in the few-shot setting.
H OW DOES G&C BEHAVE AS WE SCALE UP THE MODEL ?
People have observed that increasing the size of neural networks trained with SGD can lead to either double descent behaviorNakkiran et al. (2021) or increasing performance Kaplan et al.
This phenomenon has been previously attributed to the regularization effect of SGD. However, given the similarity of performance between G&C models and SGD-trained models, it is natural to ask whether we observe similar behaviors in G&C models as we increase the number of parameters.
Here, we show that G&C models continue to improve as we increase their size, without the use of
SGD.
To investigate this further, we conducted experiments on 2-class MNIST using G&C with varying widths of LeNet (Table 6).
Surprisingly, we found that as we increased the width of the model, its validation accuracy also increased.
This observation contradicts generalization theories based on model capacity, which suggest that increasing model size beyond a certain point leads to overfitting and reduced generalization performance.
The observation points to the hypothesis that increasing the width of a neural network can expand the volume of the good function class.
If we can identify the specific function within this class that experiences an increase in volume with more parameters, it may be possible to achieve similar benefits with a smaller model that captures the favorable properties of the function.
This could lead to more efficient and effective deep learning models that perform better without unnecessary parameter bloat.
However, further research is required to investigate this hypothesis and identify the specific functions that contribute to the observed increase in volume.
A T OY E XAMPLE : S IMPLICITY BIAS MAY ORIGINATE FROM THE VOLUME BIAS AS OPPOSED TO SGD
In this section, we study whether volume may explain simplicity bias previously observed in Shah et al.
While we have mostly measured bias in terms of generalization in this paper, we think it is a promising future direction to quantify whether other forms of bias can be attributed to the volume hypothesis instead.
One example is simplicity bias, where trained neural networks strongly prefer linear decision boundaries compared to robust ones.
We provide a toy illustration on why this may be attributed to the volume hypothesis, but leave further exploration of this as future work.
Consider the following example: a trained neural network on the slab dataset ignores the more complex y-axis, as shown on the left of Figure 3, and uses a linear decision boundary drawn along the x-axis only as opposed to the robust decision boundary shown on the right of Figure 3.
While Shah et al.
(2020) has attributed the simplicity bias in this example to SGD, we found that the simplicity bias may simply originate from the large disparity in volumes between the linear and robust functions in the loss landscape.
In fact, when we used G&C to measure the volume of the two respective decision boundaries in the parameter space, we found that the linear decision boundary has volume that is 6 orders of magnitude larger than that of the robust decision boundary.
Specifically, we estimate the volume of the solution by taking the reciprocal of the number of guesses before a solution is obtained.
The volume disparity may explain why the simple decision boundary is strongly preferred compared to the alternative.
C ONCLUSION
In this paper, we empirically show that gradient-based implicit regularization of training dynamics is not required for generalization.
Instead, we consider non-gradient optimizers that lack gradient dynamics, yet still perform well.
The strong performance of gradient-free optimizers, in particular
Guess & Check, strongly suggests that the disparate volume of good and bad hypothesis classes is the main implicit bias that enables these optimizers to succeed.
For future work, we think more critically examining the role of volume in neural network’s bias will be a fruitful and interesting direction.
R EPRODUCIBILITY S TATEMENT
We ran all the experiments with prespecified random seeds, so all of the tables will be reproducible by running the respective scripts in our included code base.
A A DDITIONAL EXPERIMENTS FOR TOY EXAMPLES
[Figure]
B I NFLUENCE OF S AMPLING R ANGE FOR G UESS & C HECK O PTIMIZER
[Table]
[Table]