Paper:
Mixed Robust/Average Submodular Partitioning: Fast Algorithms, Guarantees, and Applications
We investigate two novel mixed robust/average-case submodular data partitioning problems that we collectively call Submodular Partitioning. These problems generalize purely robust instances of the problem, namely max-min submodular fair allocation (SFA) [12] and min-max submodular load balancing (SLB) [25], and also average-case instances, that is the submodular welfare problem (SWP) [26] and submodular multiway partition (SMP) [5]. While the robust versions have been studied in the theory community [11,12,16,25,26], existing work has focused on tight approximation guarantees, and the resultant algorithms are not generally scalable to large real-world applications. This is in contrast to the average case, where most of the algorithms are scalable. In the present paper, we bridge this gap, by proposing several new algorithms (including greedy, majorization-minimization, minorization-maximization, and relaxation algorithms) that not only scale to large datasets but that also achieve theoretical approximation guarantees comparable to the state-of-the-art. We moreover provide new scalable algorithms that apply to additive combinations of the robust and average-case objectives. We show that these problems have many applications in machine learning (ML), including data partitioning and load balancing for distributed ML, data clustering, and image segmentation. We empirically demonstrate the efficacy of our algorithms on real-world problems involving data partitioning for distributed optimization (of convex and deep neural network objectives), and also purely unsupervised image segmentation.

Review:
The paper proposes new algorithms to address a set of problems falling under the umbrella term of 'submodular partitioning' -including two distinct clustering problems, namely clusteringto maximize homogeneity, or clustering so as to maximize the representation power of every cluster (e.g. so as to accelerate distributed learning). The authors consider the case where common costs are applied to each cluster (homogeneous case), or when distinct costs are applied to every cluster (hetergoeneous). Adding to this split, the authors also treat a mix of a robust and an average loss (e.g. a linear combination of the the max distortion with the average distortion when clustering).Starting from the observation that the algorithms working for the robust variants are not scalable, the authors proceed to propose (i) greedy algorithms for the robust homogeneous/heterogeneous algorithms with approximation guarantees, and (ii) a simple method for blending the solutions to the robust and the average cost optimizations.They then proceed to practical examples, demonstrating improvements over baseline algorithms for clustering. Even though the paper iswritten by experts who know how to write clearly, I found it really hard to grasp the technical part. In my understanding this is because the authors try to cover too many things (including 6 algorithms, 4 problems, and 7 theorems proved in the appendix). Even though I trust the authors know what they are doing (the appendix provides many more details, but I admit that I do not have the technical background to review them all), I think the paper would benefit from focusing on the single most interesting problem and presenting it more smoothly. The paper now reads mostly a list of pointers to other works, combinations of either existing results, or results presented in the appendix, and comes with very little explanation of the techniques being used - I was surprised that the Lovasz extension was presented.I would haveappreciated more reading (and understanding) a paper on one problem, with one algorithm, rather than a problem on 8 problems with 6 algorithms. I am also not too convinced when it comes to the evaluation of the proposed algorithms. For problems 1-3 the authors only compare to very weak baselines (random splitting). Since in all these three problems the case of lambda = 0 is considered, I would have expected that the authors either compare to the existing algorithms for robust partitioning and report either improvements in time, or in accuracy. Problem 4 is the sole case where lambda > 0 is used (which is a bit underwhelming, since treating such cases is one of the main contributions of this paper); for that problem (unsupervised segmentation) the authors again compare to weak baselines, not considering e.g. any method that uses graph-cuts. Furthermore, the authors mention in the supplemental material that their method is 'scalable' to this problem while others are not; running times are not reported, while the dependence of their method on image size is not clear. I therefore found it hard to assess how 'scalable' the method actually is. Suggestions: Apart from the broader suggestion above (to focus on a few aspects of the work), I would propose to (a) avoid coveringprior work as thoroughly, to make up space for your own contributions and (b) maybe move the 'Applications' in page 3 together with the experiments, because after the intervention of 4 pages of theory the reader has already forgotten the motivation. Minor comments: l. 259: add comma after [17] l. 304: programming -> program