Paper:
We propose a simple extension to the ReLU-family of activation functions that allows them to shift the mean activation across a layer towards zero. Combined with proper weight initialization, this alleviates the need for normalization layers. We explore the training of deep vanilla recurrent neural networks (RNNs) with up to 144 layers, and show that bipolar activation functions help learning in this setting. On the Penn Treebank and Text8 language modeling tasks we obtain competitive results, improving on the best reported results for non-gated networks. In experiments with convolutional neural networks without batch normalization, we find that bipolar activations produce a faster drop in training error, and results in a lower test error on the CIFAR-10 classification task.

Review:
This paper proposes a self-normalizing bipolar extension for the ReLU activation family. For every neuron out of two, authors propose to preserve the negative inputs. Such activation function allows to shift the mean of i.i.d. variables to zeros in the case of ReLU or to a given saturation value in the case of ELU.
Combined with variance preserving initialization scheme, authors empirically observe that the bipolar ReLU allows to better preserve the mean and variance of the activations through training compared to regular ReLU for a deep stacked RNN.
Authors evaluate their bipolar activation on PTB and Text8 using a deep stacked RNN.  They show that bipolar activations allow to train deeper RNN (up to some limit) and leads to better generalization performances compared to the ReLU /ELU activation functions. They also show that they can train deep residual network architecture on CIFAR without the use of BN.
Question:
- Which layer mean and variance are reported in Figure 2? What is the difference between the left and right plots?
- In Table 1, we observe that ReLU-RNN (and BELU-RNN for very deep stacked RNN) leads to worst validation performances. It would be nice to report the training loss to see if this is an optimization or a generalization problem.
- How does bipolar activation compare to model train with BN on CIFAR10?
- Did you try bipolar activation function for gated recurrent neural networks for LSTM or GRU?
- As stated in the text, BELU-RNN outperforms BN-LSTM for PTB. However, BN-LSTM outperforms BELU-RNN on Text8. Do you know why the trend is not consistent across datasets?
-Clarity/Quality
The paper is well written and pleasant to read
- Originality:
Self-normalizing function have been explored also in scaled ELU, however the application of self-normalizing function to RNN seems novel.
- Significance:
Activation function is still a very active research topic and self-normalizing function could potentially be impactful for RNN given that the normalization approaches (batch norm, layer norm) add a significant computational cost. In this paper, bipolar activations are used to train very deep stacked RNN. However, the stacked RNN with bipolar activation are not competitive regarding to other recurrent architectures. It is not clear what are the advantage of deep stacked RNN in that context.