  We study a fundamental transfer learning process from source to target linear
regression tasks, including overparameterized settings where there are more
learned parameters than data samples. The target task learning is addressed by
using its training data together with the parameters previously computed for
the source task. We define the target task as a linear regression optimization
with a regularization on the distance between the to-be-learned target
parameters and the already-learned source parameters. This approach can be also
interpreted as adjusting the previously learned source parameters for the
purpose of the target task, and in the case of sufficiently related tasks this
process can be perceived as fine tuning. We analytically characterize the
generalization performance of our transfer learning approach and demonstrate
its ability to resolve the peak in generalization errors in double descent
phenomena of min-norm solutions to ordinary least squares regression. Moreover,
we show that for sufficiently related tasks the optimally tuned transfer
learning approach can outperform the optimally tuned ridge regression method,
even when the true parameter vector conforms with isotropic Gaussian prior
distribution. Namely, we demonstrate that transfer learning can beat the
minimum mean square error (MMSE) solution of the individual target task.
