Paper:
We propose the dense RNN, which has fully connections from each hidden state directly to multiple preceding hidden states of all layers. As the density of the connection increases, the number of paths through which the gradient flows is increased. The magnitude of gradients is also increased, which helps to prevent the vanishing gradient problem in time. Larger gradients, however, may cause exploding gradient problem. To complement the trade-off between two problems, we propose an attention gate, which controls the amounts of gradient flows. We describe the relation between the attention gate and the gradient flows by approximation. The experiment on the language modeling using Penn Treebank corpus shows dense connections with the attention gate improve the model's performance of the RNN.

Review:
Summary: 
This paper proposes a fully connected dense RNN architecture that has connections to every layer and the preceding connections of each layer. The connections are also gated by using a simple gating mechanism. The authors very briefly discusses about the effect of these on the dynamics of the learning. They report results on PTB character-level language modelling task.
Questions:
What is the computational complexity of this approach compared to a vanilla RNN architecture?
What is the implications of these skip connections in terms of memory consumption during BPTT?
Did you use gradient clipping and have you used any specific type of initialization for the parameters?
How would this approach would compare against the Clockwork RNNs which has a block-diagonal weight matrices? [1]
How would dense-RNNs compare against to the MANNs [2]?
How would you implement this model efficiently?
Pros:
Interesting idea.
Cons:
Lack of experiments and empirical results supporting the arguments.
Hand-wavy theory.
Lack of references to the relevant literature. 
General Comments:
In general the paper is relatively well written despite having some minor typos. The idea is interesting, however the experiments in this paper is seriously lacking. The only results presented in this paper is on PTB. The results are quite behind the SOTA and PTB is a really tiny, toyish language modeling task. The theory is very hand-wavy, the connections to the previous attempts to come up with related properties of the recurrent models should be cited. The Figure 2 is very related to the Gersgorin circle theorem in [3]. The discussion about the skip-connections is very related to the results in [2]. 
Overall, I think this paper is rushed and not ready for the publication.
[1] Koutnik, J., Greff, K., Gomez, F., & Schmidhuber, J. (2014, January). A clockwork rnn. In International Conference on Machine Learning (pp. 1863-1871).
[2] Gulcehre, Caglar, Sarath Chandar, and Yoshua Bengio. "Memory Augmented Neural Networks with Wormhole Connections." arXiv preprint arXiv:1701.08718 (2017).
[3] Zilly, Julian Georg, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. "Recurrent highway networks." arXiv preprint arXiv:1607.03474 (2016).