  Recently, orthogonal recurrent neural networks (RNNs) have emerged as
state-of-the-art models for learning long-term dependencies. This class of
models mitigates the exploding and vanishing gradients problem by design. In
this work, we employ tools and insights from differential geometry to offer a
novel perspective on orthogonal RNNs. We show that orthogonal RNNs may be
viewed as optimizing in the space of divergence-free vector fields.
Specifically, based on a well-known result in differential geometry that
relates vector fields and linear operators, we prove that every divergence-free
vector field is related to a skew-symmetric matrix. Motivated by this
observation, we study a new recurrent model, which spans the entire space of
vector fields. Our method parameterizes vector fields via the directional
derivatives of scalar functions. This requires the construction of latent inner
product, gradient, and divergence operators. In comparison to state-of-the-art
orthogonal RNNs, our approach achieves comparable or better results on a
variety of benchmark tasks.
