We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks. Many problems in machine learning are best formulated using sequential data and appropriate models for these tasks must be able to capture temporal dependencies in sequences, potentially of arbitrary length. One such class of models are recurrent neural networks (RNNs), which can be considered a learnable function f whose output h t = f (x t , h t−1 ) at time t depends on input x t and the model's previous state h t−1 . Training of RNNs with backpropagation through time (Werbos, 1990) is hindered by the vanishing and exploding gradient problem Hochreiter & Schmidhuber, 1997; Bengio et al., 1994) , and as a result RNNs are in practice typically only applied in tasks where sequential dependencies span at most hundreds of time steps. Very long sequences can also make training computationally inefficient due to the fact that RNNs must be evaluated sequentially and cannot be fully parallelized. A recently proposed method for easier modeling of long-term dependencies is "attention". Attention mechanisms allow for a more direct dependence between the state of the model at different points in time. Following the definition from (Bahdanau et al., 2014) , given a model which produces a hidden state h t at each time step, attention-based models compute a "context" vector c t as the weighted mean of the state sequence h by where T is the total number of time steps in the input sequence and α tj is a weight computed at each time step t for each state h j . These context vectors are then used to compute a new state sequence s, where s t depends on s t−1 , c t and the model's output at t − 1. The weightings α ij are then computed by e tj = a(s t−1 , h j ), α tj = exp(e tj ) T k=1 exp(e tk ) where a is a learned function which can be thought of as computing a scalar importance value for h j given the value of h j and the previous state s t−1 . This formulation allows the new state sequence s to have more direct access to the entire state sequence h. Attention-based RNNs have proven effective in a variety of sequence transduction tasks, including machine translation (Bahdanau et al., 2014) , image captioning (Xu et al., 2015) , and speech recognition (Chan et al., 2015; Bahdanau et al., 2015) . Attention can be seen as analogous to the "soft addressing" mechanisms of the recently proposed Neural Turing Machine (Graves et al., 2014) and End-To-End Memory Network (Sukhbaatar et al., 2015) models. 1
Raffel and Ellis REF applied attention mechanism to the recurrent neural network (RNN) and achieved advanced results on multiple NLP tasks.
Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
ArXiv
ArXiv
Computer Science