Paper:
It is unclear whether the extensively applied long-short term memory (LSTM) is an optimised architecture for recurrent neural networks. Its complicated design makes the network hard to analyse and non-immediately clear for its utilities in real-world data. This paper studies LSTMs as systems of difference equations, and takes a theoretical mathematical approach to study consecutive transitions in network variables. Our study shows that the cell state propagation is predominantly controlled by the forget gate. Hence, we introduce DecayNets, LSTMs with monotonically decreasing forget gates, to calibrate cell state dynamics. With recurrent batch normalisation, DecayNet outperforms the previous state of the art for permuted sequential MNIST. The Decay mechanism is also beneficial for LSTMbased optimisers, and decrease optimisee neural network losses more rapidly.

Review:
This paper analyses the internal dynamics of an LSTM, focusing on the cell state as being the most important component, and analyses what directly influences the contents of the cell state using difference equations. The authors note that at any timestep, the output cell state is the sum of (previous cell state * forget) and (input gate * input). The former can only shrink or maintain the cell value, which the authors label 'catch' and the latter can increase the magnitude, labelled 'release'.
The authors show that for a single neuron, with chosen values for the forget gate and inputs, consistent growth or consistent shrinking of the cell state can be observed. When the forget is large, say 0.9, the input gate can be anywhere in the range 0.5, 1.0] and still produce growth in the cell value. For forget gate = 0.25, an even larger range of input gate values all produce a shrinking cell value.
Due to the forget gate seemingly having a stronger effect than the (input gate * input) component, the authors propose to hard wire the forget gate to produce a continuous and monotonic decrease, producing the DecayNet. The rate of this decay is controlled by a learned function of the input and previous hidden state, with some shifting in order to maintain a monotonic decrease. Each forget neuron will decrease at a different rate through the processing of a sequence, leading to sections of the cell state which will decay slowly and sections which will decay quickly.
The authors perform two sets of experiments. The second is sequence classification with the standard 'pixel by pixel' permuted sequential MNIST, in which they show a new SOTA with using Recurrent Batch Norm combined with DecayNet. They also demonstrate a DecayNet with fewer parameters producing roughy the same median performance as a normal LSTM but with lower variance.
The first experiment is described as "image classification", with MNIST and Fashion-MNIST. This section is unclear to me, I had initally assumed that the data would be fed in one pixel at a time, but due to the presence of the other experiments I presume this is not the case. It is not clear what the 'time' dimension is in how the RNNs are applied here, if not through some ordering of pixels. If the entire image is presented as a flattened input, and the time dimension is iterating through the dataset, then there is no reason to use an RNN here. More detail must be added here to make it clear exactly how these RNNs are being applied to images - the text says the softmax layer is produced from the final hidden state, but without the information about how the different hidden states are produced for a given training example this not meaningful. I can imagine that both tasks are pixel by pixel, and the only difference is whether to apply the permutation.. but that is my guesswork.
In general I find this paper an interesting idea, reasonably well communicated but some parts are not clear. All the experiments (as far as I can tell) work on fixed length sequences. One advantage of an LSTM is that can run onnline on arbitrary length data, for example when used in a RL Agent. In those circumstances, does learning a fixed monotonic delay on the forget gate make sense? I would guess not, and therefore I think the paper could be more explicit in indicating when a DecayNet is a good idea.
There are definitely tasks in which you want to have the forget gate drop to zero, to reset the state, and then go back up to 1 in subsequent timesteps to remember some new information. Presumably the monotonic delay would perform poorly.
Is DecayNet appropriate only when you have fixed length sequences, where the distribution of 'when does relevant information appear in the input' is fixed? These questions make me doubt the generality of this approach, whereas "this reformulation increases LSTM modelling power ... and also yields more consistent results" from the abstract reads like this is a strictly better LSTM. A much wider variety of experiments would be required to justify this sentence. 
It would be interesting to see some diagrams of forget gate / cell state changes throughout a real task, ie a graph with `k` on the x axis. The presentation of the new forget gate in "System 2" is clear in terms of being able to implement it, but it's not intuitive to me what this actually looks like. The graphs I suggest might go a long way to providing intuition for readers.
Overall while I like the spirit of trying to understand and manipulate LSTM learning dynamics I am recommending reject. I do not think the paper sufficiently motivates why a monotonic decay should be good, and while the new SOTA on permuted MNIST is great, I'm concerned that the first experiments are not reproducable, as detailed previously in this review. All hyperparameters appear to be present, so this paper would be reproducable, except for the NIST experiments.
General recommendations for a future resubmission:
* Clarify description of first MNIST experiments, and how they are different from permuted MNIST.
* Experiments on a wider variety of canonical RNN tasks - Penn Treebank is an obvious contender.
* Some mention of in what situations this is obviously not a good model to use (RL?)
* More intuition / visualisations as to what the internal dynamics inside DecayNet look like, vs normal LSTM.
* Devote less space to the initial dynamics analysis, or modify to be representative of a real task. This part was interesting on first read, but the only thing I think it really proves is 'when we artificially choose the input values things get bigger or smaller'. The important thing is, what actually happens when training on a task that we care about - if the same catch and release dynamics are observable, then that makes the idea more compelling.
Notes and suggestions:
I feel the notation would be clearer if instead of k = 1 .. D, this index was t = 1 ... T. This makes it cleare that s_k is not the k'th item in the array, but rather than whole activation array at a specific time. The notation \sigma_t could then be replace with \tanh.
"We replace \sigma_t with sin for an ergodic delay over time": as this is a new gate for the forget gate, should this be \sigma_s?
One DL rule of thumb heard relatively often is to simply initialise LSTM forget gate biases to 1, to "remember more by default". As this is a (much simpler) way of trying to influence the behaviour of the gate, and it anecdotally improves data efficiency, it is worth mentioning in the paper.