  Deep neural networks (DNNs) have achieved remarkable success in computer
vision; however, training DNNs for satisfactory performance remains challenging
and suffers from sensitivity to empirical selections of an optimization
algorithm for training. Stochastic gradient descent (SGD) is dominant in
training a DNN by adjusting neural network weights to minimize the DNNs loss
function. As an alternative approach, neuroevolution is more in line with an
evolutionary process and provides some key capabilities that are often
unavailable in SGD, such as the heuristic black-box search strategy based on
individual collaboration in neuroevolution. This paper proposes a novel
approach that combines the merits of both neuroevolution and SGD, enabling
evolutionary search, parallel exploration, and an effective probe for optimal
DNNs. A hierarchical cluster-based suppression algorithm is also developed to
overcome similar weight updates among individuals for improving population
diversity. We implement the proposed approach in four representative DNNs based
on four publicly-available datasets. Experiment results demonstrate that the
four DNNs optimized by the proposed approach all outperform corresponding ones
optimized by only SGD on all datasets. The performance of DNNs optimized by the
proposed approach also outperforms state-of-the-art deep networks. This work
also presents a meaningful attempt for pursuing artificial general
intelligence.
