Paper:
A CONVERGENCE ANALYSIS OF GRADIENT DESCENT FOR DEEP LINEAR NEURAL NETWORKS
We analyze speed of convergence to global optimum for gradient descent training a deep linear neural network (parameterized as x → W N W N −1 · · · W 1 x) by minimizing the 2 loss over whitened data. Convergence at a linear rate is guaranteed when the following hold: (i) dimensions of hidden layers are at least the minimum of the input and output dimensions; (ii) weight matrices at initialization are approximately balanced; and (iii) the initial loss is smaller than the loss of any rank-deficient solution. The assumptions on initialization (conditions (ii) and (iii)) are necessary, in the sense that violating any one of them may lead to convergence failure. Moreover, in the important case of output dimension 1, i.e. scalar regression, they are met, and thus convergence to global optimum holds, with constant probability under a random initialization scheme. Our results significantly extend previous analyses, e.g., of deep linear residual networks (Bartlett et al., 2018).

Review:
This paper studies the convergence of gradient descent on the squared loss of deep linear neural networks. The authors prove linear convergence rate if (1) the network dimensions are big enough so that the full product can have full rank, (2) the singular values of each weight matrices are approximately the same, (3) the initialized point is “close enough” to the target.
First of all, this paper is well-written. It reads smoothly, effectively presents the key ideas and implications of the result, and properly answers to possible concerns that arise while reading. The improvement over the previous work ([Bartlett et al 18’]) is quite substantial.
Deep linear neural networks are important, and having a good understanding of linear neural networks can provide us useful insights for understanding the more complex ones, i.e., the nonlinear neural networks. In that regard, I really liked the discussion at the end of Section 3.1. My general opinion for this paper is acceptance, but I also have a number of concerns and questions.
My main concern about the study of GD on linear neural network is whether we really get any “benefit” or “acceleration” from depth, i.e., is GD on linear neural nets any faster than GD on linear models. It’s been shown that we get acceleration in some cases (e.g., $\ell_p$ regression when $p>2$ [Arora et al. 18’]), but some other results (e.g., [Shamir 18’] mentioned in Section 5) show that GD on linear neural nets (when weight matrices are all scalar) suffer exponential (in depth) increase in convergence time at near zero region, due to the vanishing gradient phenomenon. From my understanding, this paper circumvents this problem by assuming deficiency margin, because in the setting of [Shamir 18’], deficiency margin means that the initialized product ($W_{1:N}$) has the same sign as $\Phi$ and far enough from zero, so we don’t have to pass through the near-zero region.
Even with the deficiency margin assumption, the exponential dependence in depth can also be observed in this paper, if we use independent initialization of each weight matrices. In Claim 3, in order to get the probability 0.49 result, the margin $c$ must be very small (O(1/N^N)) as N goes to infinity, resulting in very small $\delta$ and $\eta$ in Theorem 1, and convergence time $T$ exploding in depth. On the other hand, if we fix $0 < c < 1$, then the probability of satisfying deficiency margin will be smaller and smaller as $N$ increases. Is this “blow-up in N” problem due to the fact that the loss is l2? Or am I making false claims? I would like to hear the authors’ opinion about this.
The paper proposes a balanced initialization scheme that doesn’t suffer exponential blow up (Procedure 1 and Theorem 2), but even with this, the learning rate must decay to zero in polynomial rate in N, also resulting in polynomial increase in convergence time as depth increases. Moreover, this type of initialization scheme (specifically tailored for linear neural networks) is not what people would do in practice; we normally would initialize each layer at random, and may suffer the problems discussed in the above paragraph. That is why I’d love to hear about the authors’ future work on layer-wise independent initialization, as noted in the conclusion section.
Below, I’ll list specific concerns/questions/comments.
* In my opinion, the statements about “necessity” of two key assumptions are too strong, because the authors only provide counterexamples of non-convergence. As [Theorem 3, Shamir 18’] shows (although in scalar case), even when the assumptions are not satisfied, a convergence rate $O(exp(N) * log(1/\epsilon))$ is possible. It will be an interesting future work to clearly delineate the boundary between convergence and non-convergence.
* In Thm 2 and Claim 3, what happens if dimension $d_0$ is smaller? What is the reason that you had to restrict it to high dimension? Is it due to high variance with few samples?
* In Thm 2, constants $d’_0$ and $a$ hide the dependence of the result on p, but I would suggest stating the dependence of those parameters on p, and also dependence on other parameters such as N.
* In Section 5, there is a statement “This negative result, a theoretical manifestation of the “vanishing gradient problem”, is circumvented by balanced initialization.” Can you elaborate more on that? If my understanding is correct, there is still $\sigma_min$ multiplier in Eq (9), which means that at near-zero regions, the gradient will still vanish.
I appreciate the authors for their efforts, especially on the heavy math in the proof of the main theorem. I would like to hear your comments and/or corrections (especially on my “T blowing up in N” claim) and discuss further.