There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994) . In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.
In REF , a soft constraint technique is proposed to deal with the vanishing gradient in training recurrent neural networks (RNNs).
On the difficulty of training Recurrent Neural Networks
arXiv: Learning
Mathematics Computer Science