Paper:
Diverse Ensemble Evolution: Curriculum Data-Model Marriage
We study a new method "Diverse Ensemble Evolution (DivE 2 )" to train an ensemble of machine learning models that assigns data to models at each training epoch based on each model's current expertise and an intra-and inter-model diversity reward. DivE 2 schedules, over the course of training epochs, the relative importance of these characteristics; it starts by selecting easy samples for each model, and then gradually adjusts towards the models having specialized and complementary expertise on subsets of the training data, thereby encouraging high accuracy of the ensemble. We utilize an intra-model diversity term on data assigned to each model, and an inter-model diversity term on data assigned to pairs of models, to penalize both within-model and cross-model redundancy. We formulate the data-model marriage problem as a generalized bipartite matching, represented as submodular maximization subject to two matroid constraints. DivE 2 solves a sequence of continuous-combinatorial optimizations with slowly varying objectives and constraints. The combinatorial part handles the data-model marriage while the continuous part updates model parameters based on the assignments. In experiments, DivE 2 outperforms other ensemble training methods under a variety of model aggregation techniques, while also maintaining competitive efficiency.

Review:
This paper proposes a new technique for training ensembles of predictors for supervised-learning tasks. Their main insight is to train individual members of the ensemble in a manner such that they specialize on different parts of the dataset reducing redundancy amongst members and better utilizing the capacity of the individual members. The hope is that ensembles formed out of such predictors will perform better than traditional ensembling techniques.   The proposed technique explicitly enforces diversity in two ways: 1. inter-model diversity which makes individual models (predictors) different from each other and 2. intra-model diversity which makes predictors choose data points which are not all similar to each other so that they don't specialize in a very narrow region of the data distribution. This is posed as a bipartite graph matching problem which aims to find a matching between samples and models by selecting edges such that the smallest sum of edge costs is chosen (this is inverted to a maximization problem by subtracting from the highest constant cost one can have on the edges.) To avoid degenerate assignments another matching constraint is introduced which restricts the size of samples selected by each model as well. This will ensure that no model starves of samples. The two constraints are posed as matroid constraints on a utility which consists of three terms: 1. The sum of edge costs to be minimized (posed as a maximization) 2. a inter-model diversity term and 3. a intra-model diversity term. The last two terms are chosen from the family of submodular functions which are well-known to naturally model diversity and diminishing return properties.   Since the overall objective function results in a combinatorial (submodular) and a continuous optimization problem the proposed solution method alternates between the two. First the discrete optimization is solved to find a better matching and then the continuous optimization is run to train the models on the new matches.   A curriculum is designed for this process so that initially there is more weight on diversity terms and then reducing them slowly.   Experiments are presented on CIFAR10, CIFAR100, Fashion and STL10 datasets. Bagging and random initialization of DNNs are used as the training baselines. A number of inference schemes are demonstrated with the best performing ones being the Top-K NN-LossPredict which trains a NN to predict for a given sample which m models are the best ones.   On all four datasets DivE2 performs the best using mostly Top-K NN-LP as the inference method. On CIFAR-100 DivE2 + all average inference works the best.  Comments:  - The paper is generally well-written and easy to understand. Thanks!!  - The proposed method is interesting, novel and tackles a relevant problem.  - Major concerns:   1. Missing baselines: In addition to the bagging and random initialization training methods the natural other ones to try which try to incorporate diversity are [38] "Why M-heads are better than one..." and [not currently cited] "Stochastic MCL for Training Diverse Deep Ensembles, Lee et al.", NIPS 2016 and since the authors are not tackling structured prediction tasks but multi-class classification tasks: "Contextual Sequence Prediction with Application to Control Library Optimization" by Dey at al., RSS 2012 which also provides the theoretical groundings for [10].  2. Datasets and tasks: The authors are currently focussed on small multi-class datasets. It will be much stronger statement to show results on larger datasets like ImageNet and/or MS COCO or other non-vision large datasets for multi-class tasks. (Note that most of the related work in this area have demonstrated on structured prediction tasks but I understand that the scope of this paper focused on multi-class prediction problems.)  I am happy to be convinced why any/all of the above baselines are not suitable candidates for comparison to the proposed method.   3. List of relevant work which should be discussed and cited:   "Stochastic MCL for Training Diverse Deep Ensembles", Lee et al., NIPS 2016  "Contextual Sequence Prediction with Application to Control Library Optimization" by Dey at al., RSS 2012  "Learning Policies for Contextual Submodular Prediction", Ross et al. ICML 2013.   "Predicting Multiple Structured Visual Interpretations", Dey et al., ICCV 2015   "Confident Multiple Choice Learning", Lee et al., ICML 2017  - Minor concerns:  1. From the results presented in Table 2 it seems that Top-k NN-LP should always be used as the inference technique even with other training methods like BAG or RND. Is this assessment fair?   2. On CIFAR-100 all-average is performing the best although Top-K NN-LP is not that far behind. On a larger dataset like ImageNet or MS Coco will all average still do the best or do the authors think that Top-K NN-LP will prevail?