  Deep ensembles perform better than a single network thanks to the diversity
among their members. Recent approaches regularize predictions to increase
diversity; however, they also drastically decrease individual members'
performances. In this paper, we argue that learning strategies for deep
ensembles need to tackle the trade-off between ensemble diversity and
individual accuracies. Motivated by arguments from information theory and
leveraging recent advances in neural estimation of conditional mutual
information, we introduce a novel training criterion called DICE: it increases
diversity by reducing spurious correlations among features. The main idea is
that features extracted from pairs of members should only share information
useful for target class prediction without being conditionally redundant.
Therefore, besides the classification loss with information bottleneck, we
adversarially prevent features from being conditionally predictable from each
other. We manage to reduce simultaneous errors while protecting class
information. We obtain state-of-the-art accuracy results on CIFAR-10/100: for
example, an ensemble of 5 networks trained with DICE matches an ensemble of 7
networks trained independently. We further analyze the consequences on
calibration, uncertainty estimation, out-of-distribution detection and online
co-distillation.
