Recurrent neural networks (RNNs) have shown success for many sequence-modeling tasks, but learning long-term dependencies from data remains difficult. This is often attributed to the vanishing gradient problem, which shows that gradient components relating a loss at time t to time t − τ tend to decay exponentially with τ . Long short-term memory (LSTM) and gated recurrent units (GRUs), the most widely-used RNN architectures, attempt to remedy this problem by making the decay's base closer to 1. NARX RNNs 1 take an orthogonal approach: by including direct connections, or delays, from the past, NARX RNNs make the decay's exponent closer to 0. However, as introduced, NARX RNNs reduce the decay's exponent only by a factor of n d , the number of delays, and simultaneously increase computation by this same factor. We introduce a new variant of NARX RNNs, called MIxed hiSTory RNNs, which addresses these drawbacks. We show that for τ ≤ 2 n d −1 , MIST RNNs reduce the decay's worst-case exponent from τ /n d to log τ , while maintaining computational complexity that is similar to LSTM and GRUs. We compare MIST RNNs to simple RNNs, LSTM, and GRUs across 4 diverse tasks. MIST RNNs outperform all other methods in 2 cases, and in all cases are competitive.
Moreover, the MIST-RNN REF model has been developed based on a gating mechanism that is similar to GRU.
Revisiting NARX Recurrent Neural Networks for Long-Term Dependencies.
ArXiv
ArXiv
Computer Science