Paper:
Full-Capacity Unitary Recurrent Neural Networks
Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be learned? To address this question, we propose full-capacity uRNNs that optimize their recurrence matrix over all unitary matrices, leading to significantly improved performance over uRNNs that use a restricted-capacity recurrence matrix. Our contribution consists of two main components. First, we provide a theoretical argument to determine if a unitary parameterization has restricted capacity. Using this argument, we show that a recently proposed unitary parameterization has restricted capacity for hidden state dimension greater than 7. Second, we show how a complete, full-capacity unitary recurrence matrix can be optimized over the differentiable manifold of unitary matrices. The resulting multiplicative gradient step is very simple and does not require gradient clipping or learning rate adaptation. We confirm the utility of our claims by empirically evaluating our new full-capacity uRNNs on both synthetic and natural data, achieving superior performance compared to both LSTMs and the original restricted-capacity uRNNs.

Review:
This paper describes a method for training recurrent neural networks with unitary transition matrices that doesn't rely on restricting the set of unitary matrices that can be learned. The authors also provide some theoretical motivation for their method by proposing the use of Givens operators (similar to Givens rotations) to quantify the representational power of a certain type of unitary matrix. They use this mathematical machinery to show that the original parameterization of uRNNs is not able to represent the entire unitary group for N x N matrices for which N > 22. The authors propose to use a gradient descent optimization method that operates directly on the Stiefel manifold of unitary matrices and show on a couple of tasks that this method leads to results that are at least as good and sometimes clearly superior to results obtained with the original uRNN parameterization and LSTMs.
I think this is a strong paper in that it presents multiple theoretical and empirical contributions. The theoretical ideas and proposed optimization algorithm are in my eyes more impressive than the empirical work, which is also decent but could benefit from a more thorough analysis. In any case, I think it's a very nice continuation of the ideas presented in the original paper about uRNNs.  Except for some minor issues, the paper is well written in the sense that it was easy enough for me to follow the materials about Givens operators and how they can be used to quantify the representational capacity of a unitary matrix even though this specific subject matter was rather new to me.   The proofs in the supplementary material seems sound to me and are relatively simple.   The experimental section is a bit less convincing since all the tasks are either synthetic, or using real data in a setup that is still artificial. I don't have too many issues with the actual results, but the paper sometimes confused me a bit about the precise experimental designs and the interpretation of the results.  In line 206, the authors say that they investigated values of the matrix dimensionality N that are below, *at* and above the critical dimension of 22. The value 22 itself however, is not part of the set of dimensions investigated. I also would have expected a sudden increase in the performance gap between the restricted-capacity uRNN and the full-capacity uRNN after increasing N past the value 22. The difference in performance seems to greater for the lower dimension problems. I understand that it's hard to compare these results directly, because they're MSE scores for problems of different dimensionality, but a discussion about the lack of a sudden breakdown of the restricted model for N > 22 would in my opinion improve the paper.   In lines 211-212, the authors say they report the best test results over 100 epochs for several random initializations. It's not very nice to do this early stopping on the test data directly, although I suspect the results wouldn't be very difference. Knowing how many initializations where actually used and seeing standard errors or deviations in Table 2 would make this section of the paper a lot stronger.  The issue I have with the 'copy memory' task is that it's a negative results for the restricted uRNN. Looking at the original paper about uRNNs, I now know that the restricted variant can solve the task for T=500 when trained by the original authors but fails at T=1000 when trained by the authors of this paper. I'm not saying they didn't try to get the best results possible with the restricted version, but it would have been better if there was some indication of a thorough hyperparameter search for this baseline. That said, the results for the new method on this harder version of the task are still impressive and indicative of it's potential practical usefulness.  The speech task still feels a bit artificial to me as next frame prediction for speech features is not a very common application except perhaps for training generative models of speech. I'm not certain that this task requires an RNN to learn very long term dependencies ans neighboring frames from spectrograms with relatively small window sizes tend to be highly correlated. I find the results on this task somewhat surprising. The restricted uRNN seems to do surprisingly well in terms of MSE for up to 70 hidden units, while the full-capacity models always perform better on the other metrics. I wouldn't claim that the full-capacity model clearly outperforms the restricted variant here. It would also have been interesting to see the performance of much larger RNNs of 500 hidden units. Again, this task still provides evidence that the optimization method works and clearly outperforms the LSTMs for models of this size.   While it may seem I have a lot of issues with the quality of the individual experiments, I think that taken together they still provide a decent investigation of the methods presented. However, I also think that the paper should improve the discussion of the results and could benefit from some additional details that would make replication of the results easier.