Paper:
The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Padé Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness.

Review:
This paper introduces a novel parametric activation function, called the Pade Activation Unit (PAU), for use in general deep neural networks. Pade is a rational function, which is a ratio of two polynomials, and which can very well approximate any of the usually used activation functions while having only a few parameters that can be learned from data. Moreover, the authors identify five properties that an activation function should have, and either prove or empirically show that PAUs satisfy all of them, unlike some of the baselines. Additionally, since Pade approximation can have poles and be unstable, this work introduces safe PAUs, where the polynomial in the denominator is constrained to attain values greater than or equal to one. Since one of the suggested properties is that a function using a given activation function be a universal function approximator, the authors provide a sketch of a proof that PAUs do allow that. This proof applies only to the unsafe version of the PAU, and it is unclear whether it extends to the safe PAU---an issue that is not mentioned by the authors.
Furthermore, the authors propose a stochastic version of PAU with noise injected into parameters, which allows regularization. The empirical evaluation is quite extensive, and the PAU is compared against nine baselines on five different architectures (LeNet, VGG, DenseNet, ResNet, MobileNet) on four different datasets (MNIST, Fashion MNIST, CIfar10, ImageNet) for the classification task. The evaluation confirms that PAUs can match the performance of or sometimes outperform even the best baselines while the attained learning curves show that PAUs also lead to faster convergence of trained models. Finally, the authors demonstrate that (and provide intuition why) using PAUs allow for high-performing pruned models.
I recommend ACCEPTing this paper as it is well written, extensively evaluated, and provides performance improvements or at least matches the performance of the best baseline across several datasets and model architectures.
My only two suggestions for improvement are a) make the universal approximation proof tighter by making sure that it extends to the safe PAU version, and b) evaluate the proposed activation function on tasks other than just classification.