  We demonstrate the ability of hybrid regularization methods to automatically
avoid the double descent phenomenon arising in the training of random feature
models (RFM). The hallmark feature of the double descent phenomenon is a spike
in the regularization gap at the interpolation threshold, i.e. when the number
of features in the RFM equals the number of training samples. To close this
gap, the hybrid method considered in our paper combines the respective
strengths of the two most common forms of regularization: early stopping and
weight decay. The scheme does not require hyperparameter tuning as it
automatically selects the stopping iteration and weight decay hyperparameter by
using generalized cross-validation (GCV). This also avoids the necessity of a
dedicated validation set. While the benefits of hybrid methods have been
well-documented for ill-posed inverse problems, our work presents the first use
case in machine learning. To expose the need for regularization and motivate
hybrid methods, we perform detailed numerical experiments inspired by image
classification. In those examples, the hybrid scheme successfully avoids the
double descent phenomenon and yields RFMs whose generalization is comparable
with classical regularization approaches whose hyperparameters are tuned
optimally using the test data. We provide our MATLAB codes for implementing the
numerical experiments in this paper at https://github.com/EmoryMLIP/HybridRFM.
