Recycling sub-optimial Hyperparameter Optimization models to generate efficient Ensemble Deep Learning
Authors: Pierrick Pochelu, Bruno Conche, Serge G. Petiton
Keywords: Deep Learning, hyperparameter optimization, ensemble deep learning, multi-GPU
Abstract: Ensemble Deep Learning improves accuracy over a single model by combining predictions from multiple models.  It has established itself to be the core strategy for tackling the most difficult problems, like winning Kaggle challenges. Due to the lack of consensus to design a successful deep learning ensemble, we introduce Hyperband-Dijkstra, a new workflow that automatically explores neural network designs with Hyperband and efficiently combines them with Dijkstra's algorithm.  This workflow has the same training cost than standard Hyperband running except sub-optimal  solutions are stored  and  are  candidates  to  be  selected in  the  ensemble  selection  step (recycling). Next, to predict on new data, the user gives to Dijkstra the maximum number of models wanted in  the  ensemble to control the tradeoff between accuracy and inference time.  
Hyperband is a very efficient algorithm allocating exponentially more resources to the most promising configurations. It is also capable to propose diverse models due to its pure-exploration nature, which allows Dijkstra algorithm with a smart combination of diverse models to achieve a strong variance and bias reduction. The exploding number of possible combinations generated by Hyperband increases the probability that Dijkstra finds an accurate combination which fits the dataset and generalizes on new data.
The two experimentation on CIFAR100 and on our unbalanced microfossils dataset show that our new workflow generates an ensemble far more accurate than any other ensemble of any ResNet models from ResNet18 to ResNet152.