Neuroevolution methods evolve the weights of a neural network, and in some cases the topology, but little work has been done to analyze the e ect of evolving the activation functions of individual nodes on network size, an important factor when training networks with a small number of samples. In this work we extend the neuroevolution algorithm NEAT to evolve the activation function of neurons in addition to the topology and weights of the network. The size and performance of networks produced using NEAT with uniform activation in all nodes, or homogenous networks, is compared to networks which contain a mixture of activation functions, or heterogenous networks. For a number of regression and classi cation benchmarks it is shown that, (1) qualitatively di erent activation functions lead to di erent results in homogeneous networks, (2) the heterogeneous version of NEAT is able to select well performing activation functions, (3) the produced heterogeneous networks are signi cantly smaller than homogeneous networks.
HA-NEAT REF extends NEAT to evolve AFs of neurons, topology, and weights, resulting in a heterogeneous network.
Evolving parsimonious networks by mixing activation functions
GECCO '17
Computer Science