Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-theart performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.
Agostinelli et al. REF designed a form of adaptive piecewise linear (APL) activation function which is learned independently by each neuron.
Learning Activation Functions to Improve Deep Neural Networks
ICLR 2014
arXiv: Neural and Evolutionary Computing
Computer Science Mathematics