Paper:
Recurrent networks are a widely used class of neural architectures. They have, however, two shortcomings. First, it is difficult to understand what exactly they learn. Second, they tend to work poorly on sequences requiring long-term memorization, despite having this capacity in principle. We aim to address both shortcomings with a class of recurrent networks that use a stochastic state transition mechanism between cell applications. This mechanism, which we term stateregularization, makes RNNs transition between a finite set of learnable states. We show that state-regularization (a) simplifies the extraction of finite state automata modeling an RNN's state transition dynamics, and (b) forces RNNs to operate more like automata with external memory and less like finite state machines.

Review:
The paper proposes an RNN architecture inspired from deterministic pushdown automata. An RNN is extended to use soft attention at every time step to choose from several learnable centroids.
In general, the paper is well written and the proposed model is theoretically grounded. Unfortunately, the proposed approach shines only on specifically designed benchmarks. It is not a surprise that a CF can be learned by an architecture very similar to DPDA (with addition of learnable parameters). There is a number of specifically designed tasks to test long-term memorization, such as copy/addition, etc. Furthermore, RNNs are mostly used for natural language processing tasks. This paper only conducts experiments on IMDB sentiment analysis ignoring better benchmarked tasks, such as language modelling.
It is not absolutely clear why authors claim that cell is playing the role of memory. It is always possible to rewrite LSTM formulas with h' which is concatenation of hidden state h and cell c. Results on "peephole connection"-inspired SR-LSTM-p should be benchmarked against an LSTM with peephole connections.
The claim repeated several times that RNNs operate like DFAs, not DPDAs. This is an important point in the paper and should be verbalized more. Does it mean that it is easier to learn regular languages with RNNs?
While intuitive, theorems 3.1-3.2 are very vague to be theorems. Otherwise, they should be proven or provided a sketch of proof. For example, how do you formalize "state dynamics"?
The quality of writing of the related work section is worse that the rest of the paper. Authors should explore more other hidden state regularization methods. And, perhaps, give less attention to stochastic RNNs since the final version of the proposed model is not stochastic.
To summarize, this paper provides an interesting direction but lacks in terms of experimentation and global coherence of what is claimed and what is shown.
Minor points:
- Citation of Theano is missing
- Give a sentence explaining what is hidden state "drifting"
- a-priori -> a priori