Estimates of the generalization error are proved for a residual neural network with L random Fourier features layers \bar z_{\ell+1}=\bar z_\ell + \text{Re k=1^K\bar b_{\ell k}e^{ %DIFDELCMD < {\rm %%% i \ell k\bar z_\ell}+ \text{Re k=1^K\bar c_{\ell k}e^{ %DIFDELCMD < {\rm %%% i \omega'_{\ell k}\cdot x}. An optimal distribution for the frequencies (\ell k,\omega'_{\ell k}) of the random Fourier features e^{ %DIFDELCMD < {\rm %%% i \ell k\bar z_\ell} and e^{ %DIFDELCMD < {\rm %%% i \omega'_{\ell k}\cdot x} is derived. The derivation is based on the corresponding generalization error to approximate function values f(x). The generalization error turns out to be smaller than the estimate {\|\hat f\|^2_{L^1(\mathbb{R}^d)}}/{( LK )} of the generalization error for random Fourier features with one hidden layer and the same total number of nodes LK , in the case the L^\infty-norm of f is much less than the L^1-norm of its Fourier transform \hat f. This understanding of an optimal distribution for random features is used to construct a new training method for a deep residual network that shows promising results .
Estimates of the generalization error are proved for a residual neural network with L random Fourier features layers \bar z_{\ell+1}=\bar z_\ell + \mathrm{Re k=1^K\bar b_{\ell k}e^{ %DIFDELCMD < {\rm %%% \mathrm{i \ell k\bar z_\ell}+ \mathrm{Re k=1^K\bar c_{\ell k}e^{ %DIFDELCMD < {\rm %%% \mathrm{i \omega'_{\ell k}\cdot x}. An optimal distribution for the frequencies (\ell k,\omega'_{\ell k}) of the random Fourier features e^{ %DIFDELCMD < {\rm %%% \mathrm{i \ell k\bar z_\ell} and e^{ %DIFDELCMD < {\rm %%% \mathrm{i \omega'_{\ell k}\cdot x} is derived. This derivation is based on the corresponding generalization error for the approximation of the function values f(x). The generalization error turns out to be smaller than the estimate {\|\hat f\|^2_{L^1(\mathbb{R}^d)}}/{( KL )} of the generalization error for random Fourier features with one hidden layer and the same total number of nodes KL , in the case the L^\infty-norm of f is much less than the L^1-norm of its Fourier transform \hat f. This understanding of an optimal distribution for random features is used to construct a new training method for a deep residual network . Promising performance of the proposed new algorithm is demonstrated in computational experiments .
R
\text{Re
\mathrm{Re
R
i
\mathrm{i
R
\text{Re
\mathrm{Re
R
i
\mathrm{i
R
i
\mathrm{i
R
i
\mathrm{i
R
The
This
R
to approximate
for the approximation of the
R
LK
KL
R
LK
KL
R
that shows promising results
. Promising performance of the proposed new algorithm is demonstrated in computational experiments