Paper:
M-Best-Diverse Labelings for Submodular Energies and Beyond
We consider the problem of finding M best diverse solutions of energy minimization problems for graphical models. Contrary to the sequential method of Batra et al., which greedily finds one solution after another, we infer all M solutions jointly. It was shown recently that such jointly inferred labelings not only have smaller total energy but also qualitatively outperform the sequentially obtained ones. The only obstacle for using this new technique is the complexity of the corresponding inference problem, since it is considerably slower algorithm than the method of Batra et al. In this work we show that the joint inference of M best diverse solutions can be formulated as a submodular energy minimization if the original MAP-inference problem is submodular, hence fast inference techniques can be used. In addition to the theoretical results we provide practical algorithms that outperform the current state-of-the-art and can be used in both submodular and non-submodular case.

Review:
The paper considers the problem of finding best diverse solutions of energy minimisation problems with focus on submodular energy functions. The authors further present an alternative formulation of the problem as an ordering enforcing problem. Furthermore, the proposed methods are compared to other state of the art methods in experiments.Quality. The presented results are interesting and seem to be correct. However, I'm not sure of the implications of Lemma 2. By replacing the original diversity measure by the one in (15), supermodularity of the diversity measure is achieved. What effects does this replacement have? Do the characteristics of the diversity measure change? Furthermore, regarding Corollary 2, it would be interesting which other diversity measures satisfy the given condition. Regarding the experiments, the authors seem to report the best result from all M obtained labelings. Wouldn't it make sense to also report the average quality and the corresponding standard deviation? Clarity. The clarity of the manuscript is good, but could be improved at several points: At several points, e.g. in the appendix, the authors refer to "the algorithms", however it is unclear to me what exactly they refer to. In equations (11) and (15) the authors use the same notation for different quantities. It would be interesting to explicitly state how these two diversity measures are related and what are the implications for the obtained diverse labelings when using the one or the other.Originality. The paper improves results for identifying diverse solutions jointly over a recently submitted paper, especially by approaching runtime/complexity issues. While there is significant overlap, this improvements seem sufficiently interesting and original to me.Significance. The presented results could be interesting for many applications in many different fields.Further comments: * L152: "This implies that in this case" * L184: "The corresponding factor ..." * L196: "compared to" * L 237: "If E is submodular, then" * L 254: "This order can be enforced" * Table 1: State meaning of "M" again. * Provide details on how the cross-validation for selecting lambda is performed. * L404: "the intersection over union quality measure" ... what does this mean? * L430: "much smaller than" * Is there any reasonable approach to learn which diversity measures to combine? * Please carefully check the grammar of the supplement. * several places: "there holds" => "it holds"