Implicit regularization in deep learning is perceived as a tendency of gradient-based optimization to fit training data with predictors of minimal "complexity. " The fact that only some types of data give rise to generalization is understood to result from them being especially amenable to fitting with low complexity predictors. A major challenge in formalizing this intuition is to define complexity measures that are quantitative yet capture the essence of data that admits generalization. With an eye towards this challenge , we provide the first  analysis of implicit regularization in tensor factorization , equivalent to a certain non-linear neural network. We characterize the dynamics that gradient descent induces on the factorization, and establish a bias towards low tensor rank , in compliance with empirical evidence. Then, motivated by tensor rank capturing  implicit regularization of a non-linear neural network, we empirically explore it as a measure of complexity, and find that it stays extremely low when fitting standard datasets . This leads us to believe that tensor rank may pave way to explaining both implicit regularization of neural networks , and the properties of real-world data translating this implicit regularization to generalization.
Recent efforts to unravel the mystery of implicit regularization in deep learning have led to a theoretical focus on matrix factorization -- matrix completion via linear neural network. As a step further towards practical deep learning , we provide the first theoretical analysis of implicit regularization in tensor factorization -- tensor completion via certain type of non-linear neural network. We circumvent the notorious difficulty of tensor problems by adopting a dynamical systems perspective, and characterizing the evolution induced by gradient descent. The characterization suggests a form of greedy low tensor rank search, which we rigorously prove under certain conditions, and empirically demonstrate under others. Motivated by tensor rank capturing the implicit regularization of a non-linear neural network, we empirically explore it as a measure of complexity, and find that it captures the essence of datasets on which neural networks generalize . This leads us to believe that tensor rank may pave way to explaining both implicit regularization in deep learning , and the properties of real-world data translating this implicit regularization to generalization.
R
Implicit
Recent efforts to unravel the mystery of implicit
R
is perceived as a tendency of gradient-based optimization to fit training data with predictors of minimal "complexity. " The fact that only some types of data give rise to generalization is understood to result from them being especially amenable to fitting with low complexity predictors. A major challenge in formalizing this intuition is to define complexity measures that are quantitative yet capture the essence of data that admits generalization. With an eye towards this challenge
have led to a theoretical focus on matrix factorization -- matrix completion via linear neural network. As a step further towards practical deep learning
A
theoretical
R
, equivalent to a certain
-- tensor completion via certain type of
R
characterize the dynamics that gradient descent induces on the factorization, and establish a bias towards
circumvent the notorious difficulty of tensor problems by adopting a dynamical systems perspective, and characterizing the evolution induced by gradient descent. The characterization suggests a form of greedy
R
, in compliance with empirical evidence. Then, motivated
search, which we rigorously prove under certain conditions, and empirically demonstrate under others. Motivated
A
the
R
stays extremely low when fitting standard datasets
captures the essence of datasets on which neural networks generalize
R
of neural networks
in deep learning