This paper introduces Grid Long Short-Term Memory, a network of LSTM cells arranged in a multidimensional grid that can be applied to vectors, sequences or higher dimensional data such as images. The network differs from existing deep LSTM architectures in that the cells are connected between network layers as well as along the spatiotemporal dimensions of the data. It therefore provides a unified way of using LSTM for both deep and sequential computation. We apply the model to algorithmic tasks such as integer addition and determining the parity of random binary vectors. It is able to solve these problems for 15-digit integers and 250-bit vectors respectively. We then give results for three empirical tasks. We find that 2D Grid LSTM achieves 1.47 bits per character on the Wikipedia character prediction benchmark, which is state-of-the-art among neural approaches. We also observe that a two-dimensional translation model based on Grid LSTM outperforms a phrase-based reference system on a Chinese-to-English translation task, and that 3D Grid LSTM yields a near state-of-the-art error rate of 0.32% on MNIST. Long Short-Term Memory (LSTM) networks are recurrent neural networks equipped with a special gating mechanism that controls access to memory cells [20] . Since the gates can prevent the rest of the network from modifying the contents of the memory cells for multiple time steps, LSTM networks preserve signals and propagate errors for much longer than ordinary recurrent neural networks. By independently reading, writing and erasing content from the memory cells, the gates can also learn to attend to specific parts of the input signals and ignore other parts. These properties allow LSTM networks to process data with complex and separated interdependencies and to excel in a range of sequence learning domains such as speech recognition [14] , offline hand-writing recognition [15] , machine translation [35] and image-to-caption generation [36, 25] . Even for non-sequential data, the recent success of deep networks has shown that long chains of sequential computation are key to finding and exploiting complex patterns. Deep networks suffer from exactly the same problems as recurrent networks applied to long sequences: namely that information from past computations rapidly attenuates as it progresses through the chain -the vanishing gradient problem [18] -and that each layer cannot dynamically select or ignore its inputs. It therefore seems attractive to generalise the advantages of LSTM to deep computation. We extend LSTM cells to deep networks within a unified architecture. We introduce Grid LSTM, a network that is arranged in a grid of one or more dimensions. The network has LSTM cells along any or all of the dimensions of the grid. The depth dimension is treated like the other dimensions and also uses LSTM cells to communicate directly from one layer to the next. Since the number N of dimensions in the grid can easily be 2 or more, we propose a novel, robust way for modulating the N-way communication across the LSTM cells.
The Grid LSTM REF ) is another powerful architecture that can learn to multiply 15-digit decimal numbers.
Grid Long Short-Term Memory
ICLR 2016
arXiv: Neural and Evolutionary Computing
Computer Science