  We establish conditions under which gradient descent applied to fixed-width
deep networks drives the logistic loss to zero, and prove bounds on the rate of
convergence. Our analysis applies for smoothed approximations to the ReLU, such
as Swish and the Huberized ReLU, proposed in previous applied work. We provide
two sufficient conditions for convergence. The first is simply a bound on the
loss at initialization. The second is a data separation condition used in prior
analyses.
