Paper:
Consistent Multilabel Classification
Multilabel classification is rapidly developing as an important aspect of modern predictive modeling, motivating study of its theoretical aspects. To this end, we propose a framework for constructing and analyzing multilabel classification metrics which reveals novel results on a parametric form for population optimal classifiers, and additional insight into the role of label correlations. In particular, we show that for multilabel metrics constructed as instance-, micro-and macroaverages, the population optimal classifier can be decomposed into binary classifiers based on the marginal instance-conditional distribution of each label, with a weak association between labels via the threshold. Thus, our analysis extends the state of the art from a few known multilabel classification metrics such as Hamming loss, to a general framework applicable to many of the classification metrics in common use. Based on the population-optimal classifier, we propose a computationally efficient and general-purpose plug-in classification algorithm, and prove its consistency with respect to the metric of interest. Empirical results on synthetic and benchmark datasets are supportive of our theoretical findings.

Review:
SUMMARYThe authors develop a theoretically justified thresholding method for plug-in estimators in multilabel classification, based on analyzing the Bayes-optimal classifier for the given objectives. CONTRIBUTIONS: WHAT IS (ARE) THE CLAIMED CONTRIBUTION(S) OF THE PAPER? HOW SIGNIFICANT/IMPORTANT IS (ARE) THE CLAIMED CONTRIBUTION(S)?Compute the form of the Bayes-optimal multilabel for two families of objectives, and use it to construct a multilabel classification algorithms based on plug-in estimators with good empirical performance.The theoretical contributions add to the understanding of multilabel classification and the algorithms are a welcome finding. SUPPORTThe theoretical results are well supported.The algorithms are proposed in a principled way based on the theoretical results. Regarding Algorithm 1, I find the fact that the thresholds can be efficiently computed an interesting observation.The empirical evaluation seems more illustrative than descriptive of the proposed method, but validates the theoretical findings and provides favorable evidence about their impact. Nevertheless, the empirical results presented are informative because in the theoretical results the form of the bayes-optimal classifier is derived, while in practice the class of classifiers being used is more restricted than all measurable classifiers. Section 5 therefore bridges this gap. TECHNICAL QUALITYThe technical quality of the paper is good, but in this reviewer's opinion the text would benefit from some more discussion about certain topics, as described ahead.There seems to be a stronger connection between the problem studied with the chosen metrics and binary classification (rather than just binary classification being a special case of the multilabel classification). Evidence to that is the seemingly strong connection between the theoretical results presented and those of Koyejo et al. 2014 (see the detailed comments section for an example). The discussion of the theoretical results would benefit from a more careful exploration of these connections.Section 5 would be enriched by some conjectures regarding why Algorithm 1 was competitive with Macro-Thres with the macro-averaged metrics (in Table 3). It would also be interesting to see how close the thresholds of Algorithm 1 are to 1/2 where BR was competitive with the micro-averaged metrics (in Table 1). ORIGINALITYThe paper inserts itself well in the picture, and the only suggestion I can make here is for the authors to point out that their proof techniques are substantially similar to those used by Koyejo et al. 2014 -- in particular it would be helpful for anyone checking the proofs to understand where they differ and how the treatment of multilabel classification affects the proofs. CLARITYThe paper is clear and the text reads fluidly. There are a few missing articles in the text, but otherwise the work well written and well executed. FOR THE REBUTTALIf my remarks about the parallels between the authors' results and those of Koyejo et al. (2014) are pertinent, a brief comment from the authors on the topic would he useful. DETAILED COMMENTSI find the similarity between Theorem 2 in the paper and Theorem 2 by Koyejo et al. (2014) quite striking. Upon checking the proofs and seeing that they are essentially the same, I started to wonder the reason behind that. I would venture that the two theorems are, in fact, equivalent. To see that (with a finite input space) we can define $\mathcal{X}' \doteq \{ x_{ij} : x_i \in \mathcal{X}, j \in [M] \}$, $P(X' = x_{ij}) \doteq \frac{1}{M}P(X = x_i)$ and $P(Y' = 1 | X' = x_{ij}) \doteq P(Y_j = 1 | X = x_i)$. Defining the quantities $\mathrm{TP}'$, $\gamma'$ and $\mathcal{L}'$ accordingly, we get$\mathcal{L}'(f) = \frac{c_0 + c_1 \mathrm{TP}(f) + c_2 \gamma(f) }{d_0 + d_1 TP(f) + d_2 \gamma(f)}$. So we can get Theorem 2 in the paper from Theorem 2 by Koyejo et al. (2014). (Note the important influence of the decoupling over labels and the decoupling over inputs in the reduction.) A similar observation can also be made using Proposition 1 in the submission.On the other hand the macro-averaged metric seems to be solved by combining the Bayes-optimal classifiers for $M$ separate binary classification problems.I would also point out that (in my opinion) the threshold being shared over the labels is as impressive as its being shared over the inputs.I was checking the proof of Theorem 2, and I arrived at a different form for the gradient $$\frac{\partial}{\partial f(x_i)_m}\mathcal{L}(f) = \frac{\mu(x_i)(c_1 - d_1 \mathcal{L}(f))}{D_r(f)}\left( \eta_m(x_i) + \frac{c_2 - d_2 \mathcal{L}(f)}{c_1 - d_1 \mathcal{L}(f)} \right)$$ (similar to the proof of Theorem 2, but with $(c_1 - d_1 \mathcal{L}(f))$ in the numerator, not in the denominator), which can also be found in Koyejo et al. (2014). SUGGESTED FIXES[l:36] The authors need to make sure that the citation format conforms to the conference requirements. [l:78] Remove the period [l:115] $n$ should be $N$ here (make sure you check other occurrences of $n,N$) [l:133] "corresponding... corresponds" [l:159] "metric to the confusion" [l:162] $\Psi$ was overloaded a few times up to this point, and it confused me; the usage seems consistent afterwards, but I would suggest revisiting the previous definitions and adjusting them to minimize ambiguity. [l:239] "optimizing an EUM utility" [l:244] "optimal for the macro-averaged" [l:257] "learning a multi-label" [l:351] The end of a sentence seems to be missing here. [l:406] "metrics, an optimal"POST-REBUTTAL REMARKSAfter considering that the theoretical results are a special case of previously existing results, albeit interesting observations, they are not substantial contributions, from the theoretical point of view. However, they justify the proposed algorithm, which the experiments validate and which carry interesting insights.