We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the 2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the 2 maximum margin solution.
Multilayer linear CNNs were studied in REF for binary classification problem with linear separable data distribution. And they proved that the dynamics could converge to a linear predictor related to the l 2/L bridge penalty in the frequency domain.
Implicit Bias of Gradient Descent on Linear Convolutional Networks
ArXiv
ArXiv
Computer Science Mathematics