Paper:
Vanilla RNN with ReLU activation have a simple structure that is amenable to systematic dynamical systems analysis and interpretation, but they suffer from the exploding vs. vanishing gradients problem. Recent attempts to retain this simplicity while alleviating the gradient problem are based on proper initialization schemes or orthogonality/unitary constraints on the RNN's recurrence matrix, which, however, comes with limitations to its expressive power with regards to dynamical systems phenomena like chaos or multi-stability. Here, we instead suggest a regularization scheme that pushes part of the RNN's latent subspace toward a line attractor configuration that enables long short-term memory and arbitrarily slow time scales. We show that our approach excels on a number of benchmarks like the sequential MNIST or multiplication problems, and enables reconstruction of dynamical systems which harbor widely different time scales.

Review:
Overview
This paper proposes a type of regularization for recurrent networks, with the goal of encouraging particular dynamical structures (in this case, line attractors) in the dynamics of the networks. The proposed regularization penalty is only applied to a subset of the recurrent units; the motivation for this is to allow neurons not contained in the subset to learn different structures. The paper applies this regularization method on three example machine learning sequence tasks: an addition task, a multiplication task, and sequential MNIST classification, as well as on learning a 2-D dynamical system model of a bursting neuron with two different timescales.
Major comments
I have a number of serious concerns about the paper's motivation, logic, and experiments:
- First, the paper motivates the proposed regularization as a way to encourage the network to have line attractor dynamics. In particular, the paper dismisses gated architectures as not being interpretable, stating that LSTMs and GRUs "are complicated and tedious to analyze from a DS perspective." (pg 2). However, there is recent work both theoretical (https://arxiv.org/abs/1906.01005) and empirical (https://arxiv.org/abs/1907.08549) that analyzes these gated architectures from a DS perspective. In particular, these papers demonstrate that LSTMs and GRUs are perfectly capable of learning line attractors. Given that it is possible to analyze gated architectures as dynamical systems, the overall motivation of the paper is much weaker.
- The paper proposes a squared penalty on subsets of the weights in the recurrent network as a way to encourage line attractor dynamics. However, it is not clear to me that this is sufficient. In particular, unless the subset of the network that implements the line/plane attractor is completely disconnected from the rest of the network, then the overall dynamics may not contain a line attractor (the units will interact with the unregularized units). Also, the proposed regularization penalty only penalizes the diagonal elements of the A matrix to be close to 1--but shouldn't the off diagonal elements also be penalized to be close to zero?
- Moreover, the paper makes no mention of the Jacobian of these recurrent networks. The eigenvalues of the Jacobian of the recurrent networks determine the behavior of the linearized system around fixed points--specifically, eigenvalues with real part close to 1 will exhibit slow dynamics (approximate line attractors along those dimensions). It seems to me that a much more natural way of encouraging line attractor dynamics is to place a regularization penalty on the Jacobian itself (which is analytically tricky, but numerically more plausible with modern autodifferentiation software). Regardless, the authors should compare the eigenvalues of the recurrent networks' Jacobian when using their regularization method vs without it. Does the proposed regularization encourage the Jacobian of the resulting networks to have eigenvalues close to 1?
- The paper compares the proposed method with a number of vanilla RNNs with different initializations, and an LSTM. However, a critical missing baseline is simply an RNN with l2 regularization on the weights (standard regularization in the literature). This baseline is important to determine if the proposed regularization simply helps because it is an l2 penalty on the weights (note that none of the other baselines have regularization).
- The paper motivates the method as trying to study line attractor dynamics, but then does not apply them to tasks where line attractors are required. For example, the addition and multiplication tasks require discrete memories, not line attractors. The bursting neuron approximation (2D dynamical system) also does not involve a line attractor. However, there definitely exist tasks both in neuroscience (e.g. sensory integration in decision making, path integration in navigation, etc.) and in machine learning (c.f. https://arxiv.org/abs/1906.10720) that use or require line attractors. The motivation of the paper would be much better tested on these tasks.
Minor comments
- The authors comment at the beginning of page 4 that by setting A=I, W=0, and h=0, the network contains a line attractor, but the more precise language would be to state that the network contains an N-dimensional plane attractor, where N is the number of units. Typically, 'line attractor' refers to a 1-dimensional manifold of fixed points along which the system can integrate inputs, but perturbations off of the line attractor are not remembered (decay back to the line attractor).