We show that gradient descent on an unregularized logistic regression problem with separable data converges to the max-margin solution. The result generalizes also to other monotone decreasing loss functions with an infimum at infinity, and we also discuss a multi-class generalizations to the cross entropy loss. Furthermore, we show this convergence is very slow, and only logarithmic in the convergence of the loss itself. This can help explain the benefit of continuing to optimize the logistic or cross-entropy loss even after the training error is zero and the training loss is extremely small, and, as we show, even if the validation loss increases. Our methodology can also aid in understanding implicit regularization in more complex models and with other optimization methods.
One study shows that the logistic regression model and 2-layer neural networks using monotone decreasing loss functions tend to converge in the direction of the max-margin solution when using GD and SGD REF .
The Implicit Bias of Gradient Descent on Separable Data
ArXiv
ArXiv
Computer Science Mathematics