  In scientific machine learning, regression networks have been recently
applied to approximate solution maps (e.g., potential-ground state map of
Schr\"odinger equation). In this paper, we aim to reduce the generalization
error without spending more time in generating training samples. However, to
reduce the generalization error, the regression network needs to be fit on a
large number of training samples (e.g., a collection of potential-ground state
pairs). The training samples can be produced by running numerical solvers,
which takes much time in many applications. In this paper, we aim to reduce the
generalization error without spending more time in generating training samples.
Inspired by few-shot learning techniques, we develop the Multi-Level
Fine-Tuning algorithm by introducing levels of training: we first train the
regression network on samples generated at the coarsest grid and then
successively fine-tune the network on samples generated at finer grids. Within
the same amount of time, numerical solvers generate more samples on coarse
grids than on fine grids. We demonstrate a significant reduction of
generalization error in numerical experiments on challenging problems with
oscillations, discontinuities, or rough coefficients. Further analysis can be
conducted in the Neural Tangent Kernel regime and we provide practical
estimators to the generalization error. The number of training samples at
different levels can be optimized for the smallest estimated generalization
error under the constraint of budget for training data. The optimized
distribution of budget over levels provides practical guidance with theoretical
insight as in the celebrated Multi-Level Monte Carlo algorithm.
