Paper:
We introduce the "inverse square root linear unit" (ISRLU) to speed up learning in deep neural networks. ISRLU has better performance than ELU but has many of the same benefits. ISRLU and ELU have similar curves and characteristics. Both have negative values, allowing them to push mean unit activation closer to zero, and bring the normal gradient closer to the unit natural gradient, ensuring a noiserobust deactivation state, lessening the over fitting risk. The significant performance advantage of ISRLU on traditional CPUs also carry over to more efficient HW implementations on HW/SW codesign for CNNs/RNNs. In experiments with TensorFlow, ISRLU leads to faster learning and better generalization than ReLU on CNNs. This work also suggests a computationally efficient variant called the "inverse square root unit" (ISRU) which can be used for RNNs. Many RNNs use either long short-term memory (LSTM) and gated recurrent units (GRU) which are implemented with tanh and sigmoid activation functions. ISRU has less computational complexity but still has a similar curve to tanh and sigmoid.

Review:
Summary:
- The paper proposes a new activation function that looks similar to ELU but much cheaper by using the inverse square root function.
Contributions:
- The paper proposes a cheaper activation and validates it with an MNIST experiment. The paper also shows major speedup compared to ELU and TANH (unit-wise speedup).
Pros:
- The proposed function has similar behavior as ELU but 4x cheaper.
- The authors also refer us to faster ways to compute square root functions numerically, which can be of general interests to the community for efficient network designs in the future.
- The paper is clearly written and key contributions are well present.
Cons:
- Clearly, the proposed function is not faster than ReLU. In the introduction, the authors explain the motivation that ReLU needs centered activation (such as BN). But the authors also need to justify that ISRLU (or ELU) doesn’t need BN. In fact, in a recent study of ELU-ResNet (Shah et al., 2016) finds that ELU without BN leads to gradient explosion. To my knowledge, BN (at least in training time) is much more expensive than the activation function itself, so the speedup get from ISRLU may be killed by using BN in deeper networks on larger benchmarks. At inference time, all of ReLU, ELU, and ISRLU can fuse BN weights into convolution weights, so again ISRLU will not be faster than ReLU. The core question here is, whether the smoothness and centered zero property of ELU can buy us any win, compared to ReLU? I couldn’t find it based on the results presented here.
- The authors need to validate on larger datasets (e.g. CIFAR, if not ImageNet) so that their proposed methods can be widely adopted.
- The speedup is only measured on CPU. For practical usage, especially in computer vision, GPU speedup is needed to show an impact.
Conclusion:
- Based on the comments above, I recommend weak reject.
References:
- Shah, A., Shinde, S., Kadam, E., Shah, H., Shingade, S.. Deep Residual Networks with Exponential Linear Unit. In Proceedings of the Third International Symposium on Computer Vision and the Internet (VisionNet'16).