Regret Bounds for Multilabel Classiﬁcation in Sparse Label Regimes
Abstract
Multi-label classiﬁcation (MLC) has wide practical importance, but the theoretical understanding of its statistical properties is still limited.
As an attempt to ﬁll this gap, we thoroughly study upper and lower regret bounds for two canonical MLC performance measures, Hamming loss and Precision@  .
We consider two different statistical and algorithmic settings, a non-parametric setting tackled by plug-in classiﬁers à la k -nearest neighbors, and a parametric one tackled by empirical risk minimization operating on surrogate loss functions.
For both, we analyze the interplay between a natural MLC variant of the low noise assumption, widely studied in binary classiﬁcation, and the label sparsity, the latter being a natural property of large-scale MLC problems.
We show that those conditions are crucial in improving the bounds, but the way they are tangled is not obvious, and also different across the two settings.
Introduction
Multi-label classiﬁcation (MLC) is a natural generalization of binary classiﬁcation and multi-class classiﬁcation.
It can be seen as a composition of several binary problems for the same instances, or as a multi-class problem where more than one label can be assigned to a given instance.
Several applications can be naturally modeled as a multi-label problem, including image and video annotation for multimedia search [13], tagging text documents for categorization of Wikipedia articles [9], recommendation of bid words for online ads [26] and beyond.
MLC is a well-established sub-ﬁeld of supervised learning, that has generated a tremendous interest, mainly of experimental nature, introducing a lot of interesting and diverse approaches (for example, [10, 25, 17, 33, 32, 19] ).
They have also been described in multiple survey papers across years, e.g., [29, 4].
The main concerns in
MLC research are label dependencies (marginal and conditional, see the discussion in [12]), label sparsity [18, 33], and minimization of a wide spectrum of performance metrics.
Interestingly, the theoretical understanding of MLC learning problems is still limited.
This might be due to the daunting presence of multiple and diverse loss functions that can be deﬁned over subsets of labels.
Popular loss functions include Hamming loss, Precision@  , or subset 0/1 loss.
Each such MLC loss function has its pros and cons, and there is no consensus among researchers what is the right performance metric to be used [12, 20], as the choice is certainly domain speciﬁc.
The theoretical results obtained so far mainly deal with the analysis of Bayes optimal classiﬁers for the wide variety of MLC loss functions [12, 30, 23, 24], Fisher-like consistency results [11, 16], or surrogate regret bounds [22, 32].
Results concerning generalization bounds have also been published in [6, 33, 31].
We ﬁll the gap in the landscape of theoretical results by providing upper and lower ﬁnite-sample regret bounds in MLC with a focus on computational efﬁciency.
We consider two learning setups, a nonparametric and a parametric one.
Within the former, we focus speciﬁcally on a k -nearest neighbor ( k -NN) classiﬁer, because of its attractive properties in MLC settings.
Beyond computational efﬁciency, a main reason why k -NN is a convenient approach is that it works “out of the box", operating only in the feature space, independent of the number of labels, hence making it a very versatile tool for MLC tasks.
It has also been studied in MLC, mainly empirically, for example in [34].
In the parametric setting, we introduce a MLC counterpart to the surrogate loss regret reductions introduced in [3], thereby considering empirical risk minimization solutions applied to convex surrogate loss functions.
We restrict our attention to Hamming loss and Precision@  , and thoroughly investigate the interplay between two natural assumptions, the low noise condition and the label sparsity, in determining rates of convergence.
Our results build on a number of former contributions.
The analysis of k -NN classiﬁers has been motivated by the results in [28, 27].
The proof techniques we rely upon in deriving regret bounds are related to those adopted for high probability regret bounds in binary classiﬁcation [5].
To obtain lower bounds we follow to some extent results of [2], as the proof uses reduction of a sparse MLC problem to binary classiﬁcation.
The most similar results to ours have been published in [8, 7].
In [8], the authors analyze a modiﬁed version of Hamming loss which assigns different weights to relevant and irrelevant labels.
The paper also assumes sparsity, but that assumption is stronger than ours.
The guarantees obtained are in terms of generalization and regret bounds.
[7] does not directly use Precision@  , but the setting analyzed there is similar to the one for Precision@  .
That paper also formulates a margin condition similar to ours.
Yet, the regret bounds contained there are not of a high probability type, but only bounds in expectation, similar to [2].
Moreover, [7] does not contain (matching) lower bounds.
Our results for the parametric case build on [3], which contains a regret analysis, also under low noise condition, for surrogate losses in binary classiﬁcation.
Our generalization takes additionally multi-label sparsity into account.
Main results.
We consider novel label-wise margin (low-noise) conditions that are sufﬁcient to get fast rates.
In addition, we assume that at most s out of all m labels can have conditional probability arbitrarily close to 1, whereas for the rest of the labels this probability is upper bounded by a small constant.
Such label sparsity assumption is essentially motivated by real-world multi-label problems with a large number of potential labels, where only a few such labels are assigned to single instances, a paradigmatic example being image or text annotation.
In the non-parametric setting, we consider a MLC variant of k -NN as our method of choice, since this method is well-understood in the binary case, and can be shown to achieve fast rates for 0/1-loss (which is in fact the binary counterpart to Hamming loss).
Under multi-label margin and sparsity assumptions, we present a high probability bound for the Hamming loss regret of k -NN which is of the form O
[Equation]
+  , where s is the sparsity parameter,  is the smoothness of the regression function with respect to the inﬁnity norm k .
k 1 , ↵ is the exponent in the margin condition, and  is the desired conﬁdence level.
In the special case where no sparsity is assumed we have s = m , hence the bound boils down to a more "standard" upper bound for the
Hamming loss regret.
We would like to emphasize that applying a binary regret analysis, such as the one in [5], to each of the m labels independently cannot leverage global properties like label sparsity.
Thus, in order to take advantage of label sparsity, a more involved regret analysis is needed.
Also observe that, by making the margin condition more favorable (that is, by increasing ↵ ), the dependence on the sparsity ratio s/m remains unaffected.
In case of Precision @  , we do not need label sparsity: a margin condition, that controls the gap between the  -th and the (  + 1 )-th largest conditional distribution, is by itself enough to obtain regret bounds that scale gracefully with the number of labels.
Under this assumption, we obtain a regret bound for the multi-label k -NN classiﬁer of the form (log( m/  ) /n )
(  (1+  )) / (2  +1)
+  , where  is the margin exponent for the  -th and (  + 1 )-th
largest conditional distribution.
In the parametric setting, we consider regret bounds for function classes with ﬁnite VC dimension.
Under the margin and sparsity assumptions, we relate the Hamming loss regret Reg ` H of an arbitrary
MLC classiﬁer to its MLC surrogate loss regret Reg ` via bounds of the form
Reg
[Equation]
⌘
[Equation]
where ⌫ is a parameter related to the curvature of the surrogate loss ` (e.g., ⌫ = 2 for both truncated square and exponential losses).
In turn, these bounds can be used in combination with standard uniform convergence results to control Reg ` , e.g.,
Reg ` = O ✓ d + log( m/  ) p n + a ◆ , where d is the pseudo-dimension of the VC class under consideration, and a is its approximation error.
Regret bounds for Precision@  are reduced to those for Hamming loss.
As a last contribution, we derive MLC regret lower bounds for our MLC setups revealing that, at least in the non-parametric case, our upper bound for Hamming loss is optimal up to a log s factor, and that our regret upper bound for Precision @  is optimal up to a log m factor.
Preliminaries and notation
We assume to have a feature space X endowed with a metric ⇢ X , and a set of m labels.
We denote by
Y :
= { 0 , 1 } m the set of binary vectors encoding the set of all 2 m possible label combinations.
We shall assume the existence of a pair of random variables X 2 X
and Y 2 Y with joint distribution P .
Let
M ( X , Y ) denote the set of all classiﬁers , which are measurable mappings  : X !
Y .
The goal of the learner is to obtain a classiﬁer  2 M ( X , Y ) such that  ( x ) ⇡ y for typical pairs ( x , y )
⇠ P .
The precise sense in which “  ( x ) ⇡ y ” is provided by a loss function ` : Y ⇥ Y !
[0 , 1 ) , with the interpretation that ` (˜ y , y ) measures the cost incurred when the learner predicts ˜
y while the true (multi-)label is y .
The problem of multi-label classiﬁcation can be deﬁned as ﬁnding a (vector-valued) classiﬁer  ( x ) =
(  j ( x ))
j 2 [ m ] , that minimizes the expected loss (or risk ):
[Equation]
the expectation being over X ⇥ Y ⇠ P .
For a given loss function ` , the optimal classiﬁer, often called the Bayes (optimal) classiﬁer is deﬁned as  ⇤ ` = argmin  2 M (
X , Y ) R ` (  ) .
When ` is clear from the surrounding context, we shall omit the subscript and simply write  ⇤ .
We can measure the statistical performance of a given classiﬁer  in terms of its regret , henceforth denoted by Reg ` (  ) , quantifying the sub-optimality of  .
Speciﬁcally, Reg ` (  ) measures the extent to which the expected loss of  exceeds the expected loss of the Bayes classiﬁer:
[Equation]
From a purely statistical standpoint, the goal of learning is then ﬁnding a classiﬁer  with regret as small as possible, ideally equal to zero.
Computing  ⇤ requires prior knowledge of the data distribution P .
Yet, P is assumed to be unknown and accessible only through a dataset (or statistical sample )
D
= { ( X i , Y i ) } i 2 [ n ] , where the pairs ( X i , Y i ) are drawn i.i.d.
from P .
In what follows, we denote by µ the marginal distribution of
P over X , so that µ ( A ) =
P ( X 2 A ) .
Moreover, we deﬁne the multi-label regression function ⌘ : X !
[0 , 1] m by ⌘ ( x ) =
( ⌘ j ( x ))
j 2 [ m ] , whose j -th component ⌘ j ( x ) is the conditional probability of the j -th label being one: ⌘ j ( x ) =
P ( Y j = 1 | X = x ) .
Losses for multi-label classiﬁcation
We focus on two widely used loss functions for MLC, namely Hamming loss and Precision@  , which we now deﬁne.
The Hamming loss ` H : Y ⇥ Y !
[0 , 1] is deﬁned for pairs y
=
( y j )
j 2 [ m ] 2 Y
and ˆ y
=
(ˆ y j )
j 2 [ m ] 2 Y by ` H (ˆ y , y ) :
= 1 m
X j 2 [ m ]
I { ˆ y j 6 = y j } .
Hence, ` H (ˆ y , y ) simply counts the fraction of mispredicted labels.
We observe that for all
M ( X , Y ) we have
[Equation]
implying that
⇤ ` H : X !
Y , deﬁned by
[Equation]
` H
[Equation]
is the Bayes optimal classiﬁer for ` H .
Moreover, it is not hard to see that for all  2 M ( X , Y ) we have
[Equation]
To deﬁne the Precision@  loss, we need to restrict the predicted vectors ˆ y to lie in Y  = { y 2 Y : k y k 1 =  } .
Then we have
[Equation]
where T (ˆ y ) =
{ j 2 [ m ] : ˆ y j = 1 } is the set of labels output by the predictor (corresponding to the labels of value one).
Let   ( f ) : R m !
Y  be a mapping from a vector of real-valued scores to a binary vector with  ones.
Then, the corresponding risk is:
[Equation]
Based on this, one can obtain the Bayes optimal classiﬁer  ⇤ ` @  for ` @  as one that predicts on the given x in input the  classes with the highest conditional probabilities ⌘ j ( x ) as positives, i.e  ⇤
[Equation]

[Equation]
x ) :=
[Equation]
where T  : R m !
[ m ]  returns the set of  indices that are largest for a real-valued input vector.
The regret of any classiﬁer  can be written as
[Equation]
Sparsity and margin condition
We introduce two assumptions which are fairly natural for MLC scenarios.
The ﬁrst one encodes the intuition that in typical MLC settings, when the total number of classes/labels m is large, only very few labels are active for any given region in the feature space, that is, each item x comes with a sparse label vector y .
Assumption 4.1 (Sparsity assumption) .
Deﬁne A ( s, t ) as the set of t -approximately s -sparse vectors as follows:
[Equation]
Take t 2 [0 , 1 / 2) and s 2 N .
We shall say that the t -approximate s -sparsity assumption holds if the multi-label regression function ⌘ ( x ) 2 A ( s, t ) for all x 2 X .
Assumption 4.1 always holds with s = m and t = 0 , but for many distributions encountered in practice we expect it to hold with a much smaller value of s , possibly coupled with t > 0 .
In fact, this assumption generalizes the standard sparsity assumption of “at most s relevant labels per instance" by adding a term for a long tail.
Yet, it does not introduce a hard constraint, since it applies to
⌘ ( x ) .
According to this sparsity assumption, the expected number of labels per instance x is at most
E [
P mj =
[Equation]
Y j | X = x ]  s + ( m  s ) ·
t. On real world multi-label datasets with large label spaces, the expected number of labels is usually less than 100 , while the number of possible labels m can be three or four orders of magnitude bigger [1, 21, 26].
The next assumption is a MLC variant of Tsybakov’s margin (or low noise) assumption
(e.g., [2]).
It ties up "noise" present on different labels to occur simultaneously on those labels.
Assumption 4.2 (Margin assumption) .
Let ↵  0 and C ↵  1 be constants.
We shall say that the margin assumption holds with exponent ↵ and constant C ↵ if the following holds for all ✏ 2 (0 , 1] :
[Equation]
Non-parametric setting 5.1 The k -NN regression estimator
Given x 2 X , we let { ⌧ n,q ( x ) } q 2 [ n ] be an enumeration of [ n ] such that for each q 2 [ n  1] ,
[Equation]
In words, for given x 2 X , { ⌧ n,i ( x ) } i 2 [ n ] sorts { X 1 , . . .
X n } in increasing order of ⇢ X -distance to x .
The k -nearest neighbor regression estimator ˆ ⌘ n,k : X ! [0 , 1] m is given by ˆ ⌘ n,k ( x ) :=
(ˆ ⌘ n,kj ( x ))
j 2 [ m ] where
[Equation]
The plug-in multi-label classiﬁer for Hamming loss that is based on the k -NN regression function ˆ ⌘ n,k is denoted by  n,k ` H , and deﬁned as
[Equation]
Similarly, the plug-in multi-label classiﬁer for Precision @  loss that is based on the k -NN regression function is denoted by  n,k ` @  and deﬁned as
[Equation]
In this paper, we shall scrutinize the performance of these k -NN classiﬁers in terms of regret under some assumptions on the classiﬁcation problem at hand.
Given x 2 X and r > 0 , let B r ( x ) denote the open metric ball of radius r , centered at x .
Assumption 5.1 (Smoothness assumption) .
Suppose we have an exponent  2 (0 , 1] , a constant
C  > 0 and a semi-norm k · k on R m .
We shall say that the measure-smoothness assumption holds with parameters  , C  , k · k if the following holds for all x 0 , x 1 2 X :
[Equation]
This kind of assumption is typical for k -NN.
It says that the regression function ⌘ ( x ) does not change much in the neighborhood of every input x .
This is essentially needed for k -NN convergence.
The speciﬁc version adopted here is from [5], and was further employed, e.g., in [28].
Note that, more generally, a universal rate of convergence cannot be established without assuming regularity properties of the data distribution, such as smoothness via absolute continuity.
Results of that kind are beyond the scope of this study.
We refer the reader to [14] for details on L 1 consistency and its connection to the rate of convergence.
5.2 Regret upper bound for Hamming loss
This multi-label margin assumption is a sufﬁcient condition for fast rate O (1 /n ) .
Its binary classiﬁcation counterpart is well-understood in the binary classiﬁcation literature.
We have the following high probability upper bound for the regret in terms of Hamming loss for the k -NN classiﬁer  n,k ` H , which is the main result of this section.
Theorem 5.2.
Let Assumption 5.1 hold with parameters  , C  > 0 , k · k 1 , Assumption 4.1 hold with parameters s 2 N , t 2 [0 , 1 / 2) and Assumption 4.2 hold with parameters ↵ , C ↵ > 0 .
Assume further that k in k -NN satisﬁes 16(1  2 t )  2 log(2 m/  )  k  ((1  2 t ) / (8 C  )) 1 /  ( n/ 2) .
Then, with probability at least 1   over D , the regret of the k -NN plug-in classiﬁer  n,k ` H obtained from (3) satisﬁes
[Equation]
where
[Equation]
k .
In particular with k = ⌦
[Equation]
and
[Equation]
⌘ we have
[Equation]
A few remarks are in order at this point.
Remark 5.3 .
(On the assumptions)
The regret bound given in Theorem 5.2 requires three assumptions.
The smoothness assumption of the regression function (Assumption 5.1) is natural in the sense that a universal rate of convergence cannot be established without assuming regularity properties of the data distribution (e.g., [14, 15]).
Analyzing the impact of different smoothness criteria is beyond the scope of this study.
Assumption 4.1 ensures label sparsity which allows k -NN learning to achieve regret of order s/m .
Again, this assumption is also well motivated by practice, since most of the
MLC task of practical relevance have sparse labels.
Finally, Assumption 4.2 ensures fast rates, i.e., faster than 1 / p n .
This assumption is commonly used and well-understood in the binary case.
Here, we consider a natural extension to the multi-label setup which quantiﬁes the probability of observing instances whose conditional probabilities are close to 1 / 2 .
Remark 5.4 .
(On MLC bounds based on binary classiﬁcation) Theorem 5.2 with s = m results in a high probability upper bound of the form O
[Equation]
.
This bound coincides with the fast rate upper bound which we would achieve had we applied the regret upper bound for the binary case (e.g., Theorem 4 in [5]) to each conditional ⌘ j ( x ) independently, after setting  /m and taking a union bound.
Notice that the binary result can be applied to our setup to each ⌘ j ( x ) individually because our smoothness condition is based on k .
k 1 , so that the smoothness deﬁnition for the binary regression function in [5] readily applies to our MLC setting.
Yet, we would like to stress that this need not imply that the bound so obtained is optimal.
Rather, this way of viewing MLC as a set of independent binary problems sheds light on how to construct a set of problem instances on which we can prove lower bounds based on the binary result (see Section 5.4 for details).
On the other hand, the sparsity condition s < m imposes global constraints on the MLC task that makes the bounds contained in Theorem 5.2 not readily obtainable by simply reducing to m independent binary problems.
Remark 5.5 .
(On fast and ultra fast rates)
When ↵ > 1 / 2 , the exponent of (1 /n ) log(4 s/  ) in the bound of Theorem 5.2 becomes larger than 1/2, giving a fast rate guarantee.
Similarly, when  ( ↵  1) > 1 , the exponent becomes larger than 1, and we obtain ultra fast rates.
5.3 Margin condition for Precision@ 
The margin condition for Hamming loss ensures a gap of conditional probabilities around 1 / 2 with a certain probability, which is motivated by the fact that those are the hard-to-classify instances when the target loss is the Hamming loss.
Starting from this observation, one can come up with a margin condition for Precision @  of a similar ﬂavor.
Let us denote the i th largest margin given x by ⌘ ( i ) ( x ) .
We assume there is a gap with some quantiﬁable probability between the  -th and (  + 1) -th conditional distribution, as speciﬁed next.
Assumption 5.6 (Margin condition for Precision@  ) .
There exist   0 and K   0 such that for all ⇠ > 0 we have
[Equation]
This assumption bears close resemblance to the margin condition introduced in [2] but, clearly enough, in order to make the optimal decision for Precision @  , the classiﬁer needs to distinguish between ⌘ (  ) ( x ) and ⌘ (  +1)
( x ) with a certain quantiﬁable probability over X .
With this assumption at hand, one can show the following high probability regret bound for  n,k ` @  .
Theorem 5.7.
Let Assumption 5.1 hold with parameters  , C  > 0 , k · k 1 , and Assumption 5.6 hold with parameter  > 0 and K  .
Suppose further that k satisﬁes 4 log(2 /  )  k  n/ 2 .
Then, with probability at least 1   over D it holds that
[Equation]
where  ( n, k,  ) =
C
[Equation]
In particular with
[Equation]
, we have
[Equation]
O@ ✓ log( m/  ) n ◆  (1+  ) 2  +
A + 2  .
Remark 5.8 .
Theorem 5.7 does not require label sparsity in order to achieve a logarithmic dependence on the number of labels m .
Label sparsity given in Assumption 4.1 does not help in general to optimize Precision @  , since when all conditionals are smaller than t , sparsity does not help to ﬁnd the labels with the top-  marginals, which is needed to make an optimal decision for Precision @  .
Remark 5.9 .
The margin condition in Assumption 5.6 is generally inevitable in order to obtain fast rates for Precision @  .
To see this, one should delve into the proof of the lower bound in Theorem 5.11 (in Appendix D), where it is shown that, for the multi-label problem constructed therein, the margin condition in Assumption 4.2 implies the one in Assumption 5.6 with  = ↵ .
5.4 Regret lower bounds
In this section, we present lower bounds for MLC problems with Hamming loss and Precision @  that scale with the label sparsity.
In order to do so, we introduce two reductions that map binary data to multi-label data and multi-label classiﬁers to binary classiﬁers.
These reductions are designed so as to preserve regret (scaled with sparsity for Hamming loss), in that lower bounds for binary classiﬁcation turn to lower bounds for the multi-label problem with Hamming loss and Precision @  .
Theorem 5.10.
Let us denote by P the class of probability distributions over X ⇥ Y for which the following assumptions hold: 1) Assumption 1 holds with
some
parameters  , C  > 0 , k · k 1 ; 2) Assumption 4. holds with parameters s 2 N , t 2 [0 , 1 / 2) ;
Assumption 4.
holds
with
some parameters ↵ , C ↵ > 0 .
Then there exists a constant C > 0 such that for any n  1 and any mapping ˆ f n : ( X ⇥ Y ) n 7!
M ( X , Y ) it holds that
[Equation]
where s denotes the sparsity ratio s = s/m .
Lower bound for Precision@  can be also obtained as follows.
Theorem 5.11.
Let us denote by P the class of probability distributions over X ⇥ Y for which the following assumptions hold: 1) Assumption 5.1 holds with some parameters  , C  > 0 , k · k 1 ; 2) Assumption 5.6 holds with some parameters   0 and K   0 .
Then there exists a constant C >such that for any n  1 and any mapping ˆ f n : ( X ⇥ Y ) n 7!
M ( X , Y ) it holds that
[Equation]
Notice that some bound r (  ) on the regret that holds with probability (1   ) , translates into a bound r (  ) +  on the expected regret in a straightforward manner.
Therefore any lower bound on the expected regret also applies when considering high probability bounds.
As a consequence, the
Hamming loss upper bound in Theorem 5.2 is optimal up to a log s/  factor and the regret bound for Precision @  given in Theorem 5.7 is optimal up to a log m/  factor.
It is an interesting open question whether these log factors can be eliminated to get matching regret bound.
Parametric Setting
In this section, we work with multi-label scoring functions in the form of f : X 7!
R m .
A multi-label scoring function is mapped to a multi-label classiﬁer as  f ( x ) =
( sgn ( f j ( x )))
j 2 [ m ] .
Notice that, for convenience, we now view the multi-label space as {± 1 } m , instead of { 0 , 1 } m .
Likewise the Hamming loss and Precision@  will be deﬁned on {± 1 } m .
We will work with convex surrogate loss functions with a single argument that applies to a multi-label score function as ( ` ( y i f ( x ))
j 2 [ m ] , resulting in a vector of losses for x .
The risk of a scoring function f in terms of surrogate loss ` is then deﬁned as R ` ( f ) = 1 /m
P mj =1 E ` ( Y j f j ( X )) , and the Bayes ` -risk as
R ⇤ ` = inf f R ` ( f ) , the inﬁmum being taken over all measurable functions.
The ` -regret of f is as usual Reg ` ( f )
= R ` ( f )  R ⇤ ` .
In addition, we will assume that the loss functions ` are classiﬁcation calibrated, as recalled in the next deﬁnition.
Deﬁnition 6.1.
A loss function ` : R 7!
R is classiﬁcation calibrated if for all ⌘ 2 [0 , 1] such that ⌘ 6 = 1 / 2 it holds that
[Equation]
In words, a loss function ` is classiﬁcation-calibrated if the sign of the argmin classiﬁer always equals the sign of the Bayes optimal classiﬁer according to true underlying distribution.
Examples of classiﬁcation-calibrated losses are the exponential loss, the (truncated) squared loss, and the hinge loss.
Our analysis of the regret of Hamming loss and Precision@  extends the work of [3] to MLC.
In [3], it is shown that for each classiﬁcation-calibrated loss, there exists a nondecreasing function : [0 , 1] 7!
[0 , 1 ) that provides an upper bound on the regret of the 0 - 1 loss in terms of the regret of the surrogate loss, i.e., ( Reg 0 - 1 ( f ))
 Reg ` ( f ) for all measurable function f : X 7!
R .
We extend their 0 - 1 loss result to Hamming loss and Precision@  under sparsity assumptions.
Further results, including consistency, are contained in Appendix E.
6.1 Regret upper bound for Hamming loss
We now give regret upper bounds for Hamming loss using classiﬁcation calibrated losses.
These upper bounds scale with the sparsity factor s = s/m , and apply to multi-label classiﬁers whose regret is small enough.
Notice that placing small regret constraints on the multi-label classiﬁer is unavoidable, since a classiﬁer that has very poor performance, e.g. f bad j ( x ) =
⌘ j ( x ) for all j , will always suffer a regret that does not depend on s .
Observe that a similar assumption on the performance of the classiﬁers is also implicitly made in Theorem 5.2 in the form of a lower bound ⌦ (log(2 m/  ) / (1  2 t ) 2 ) on the sample size n , which ensures that the plug-in estimator cannot have arbitrarily large regret.
[Figure]
Theorem 6.2.
Let Assumption 4.1 hold with parameters s 2 N , t 2 [0 , 1 / 2) and Assumption 4.
hold with parameters 3 ↵ and C ↵ .
If Reg ` H (  f ) < s (2  (1  2 t ))
↵ +1 then
[Equation]
where s = sm and  = C 1 / ( ↵ +1) ↵
[Equation]
To see how this regret bound depends on s , assume ( ✓ ) is of the form ( ✓ )
= ✓ ⌫ , where ⌫ > 1 .
This implies that is a strictly convex function.
With this in hand, and solving for Reg ` H (  f ) , Eq.
(4) becomes
[Equation]
Thus the Hamming loss regret is upper bounded by a power of Reg ` ( f ) , multiplied by a sparsity factor of the form s ⌫  1 ↵ + ⌫ .
For the sake of illustration, we can instantiate this upper bound for speciﬁc losses:
• The truncated quadratic loss ` ( z ) =
(max { 0 , 1  z } ) 2 is obtained with ⌫ = 2 (see [3]).
The regret in terms of this surrogate loss boils down to O ( s 1 ↵ +2 Reg ` ( f ) ↵ +1 ↵ +2 ) .
So without margin condition, that is when ↵ = 0 , this upper bound becomes O ( p s Reg ` ( f )) .
In contrast to Theorem 5.2, the dependency on sparsity ratio s here is worse by a p s factor.
Moreover, this dependency becomes less and less relevant as ↵ increases.
Very similar conclusions can be drawn for the exponential loss ` ( z ) =
e  z , since the corresponding ( ✓ ) =
p 1  ✓(see again [3]) can be lower bounded by ✓ 2 / 2 and upper bounded by ✓ 2 .
• Arbitrary ⌫ > 1 : More generally, the exponents of the sparsity and regret factors are related to the curvature of the loss as O ( s 1  1 ⌫ Reg ` ( f ) 1 ⌫ ) when no margin condition is imposed.
Remark 6.3 .
In order to apply, Theorem 6.2 needs the regret of  f be below a certain threshold, i.e.,
Reg ` H (  f ) < s (2  (1  2 t ))
↵ +1 .
This condition can be easily met by requiring  f to be an s sparse classiﬁer, and that parameter t in Assumption 4.1 is not too large.
Since it is usually the case that the classiﬁers employed in MLC are sparse, Theorem 6.2 applies to most interesting MLC problems.
To see this in detail, let Assumption 4.1 hold, and consider an s sparse classiﬁer  f .
The maximum regret w.r.t.
` H is clearly at most 2 s .
Thus, when
[Equation]
any s sparse classiﬁer will have regret Reg ` H (  f ) less then the threshold required by Theorem 6.2.
Also, notice that  = ( C ↵ ) 1 / ( ↵ +1) (1 + ↵ ) / ↵ ↵ / (1+ ↵ ) therein depends on C ↵ and ↵ .
Therefore, for an s sparse classiﬁer, (6) is satisﬁed when t is small enough, the maximum value of t depending on ↵ and C ↵ , as exempliﬁed in Figure 1.
Remark 6.4 .
In Appendix H we show that the upper bound of Theorem 6.2 cannot be improved by more than a factor of 2 .
Remark 6.5 .
When working with classes F with ﬁnite pseudo-dimension (e.g., [15]), the regret
Reg ` ( f ) of any f 2 F can as usual be decomposed into an estimation error R ` ( f )  R ` ( f ⇤ ) and an approximation error R ` ( f ⇤ )
R
⇤ ` , where f ⇤ 2 argmin f 2 F R ` ( f ) .
If each component F i of
F has ﬁnite pseudo-dimension d i , and ˆ f is the multi-label scoring function that minimizes the multi-label empirical risk minimization w.r.t.
the surrogate loss function, then the estimation error
R ` ( ˆ f )  R ` ( f ⇤ ) can with probability at least 1   be upper bounded by an expression of the form q (max i d i )+log
[Equation]
m/
[Equation]
n .
Combined, e.g., with (5), this yields regret upper bounds of the form
[Equation]
6.2 Regret upper bound for Precision@ 
Recall that Precision@  is a ranking metric that ignores the values of the scoring function components at hand, only the ranking of these values is relevant.
For instance, unlike risk R ` ( f ) , shifting the value of f ’s components by the same amount does not change R ` @  ( T  ( f )) .
As a consequence, low
Reg ` @  ( T  ( f )) does not imply low Reg ` ( f ) .
Thus, to remedy this problem, we re-center the values of the scoring function f so that f [  ] ( x ) > 0 and f [  +1] ( x ) < 0 to make it invariant to shifting values.
We denote by f 0 the centered version of f , i.e., f 0 ( x ) =
f j ( x )  (1 / 2)  f [  ] ( x ) + f [  +1]( x )
j 2 [ m ] .
Notice that Reg ` @  ( T  ( f )) =
Reg ` @  ( T  ( f 0 )) .
For simplicity, we assume that 8 x 2 X , f [  ] ( x ) 6 = f [  +1] ( x ) to remove ties.
(One simple way to avoid ties is to add inﬁnitesimal random noise to the values of f ).
At this point, we can rely on Theorem 6.2 to obtain an upper bound on Reg ` @  ( T  ( f )) in terms of the surrogate loss regret Reg ` ( f ) , when combined with the following result.
Theorem 6.6.
Let 0  s   .
If Assumption 4.1 holds with parameters s 2 N , t 2 [0 , 1 / 2) , then
[Equation]
where f 0 ( x ) =
f j ( x )  (1 / 2)  f [  ] ( x ) + f [  +1]( x )
j 2 [ m ] .
Equivalently, for f that satisﬁes, 8 x , f [  ] ( x ) > 0 , f [  +1] ( x ) < 0 ,
[Equation]
The inequalities are equalities if 8 x 2 X , 8 k 2 [  ] , ⌘ [ k ] ( x ) > 12 .
As an example, when a truncated quadratic loss is used, we obtain following upper bound by combining Theorem 6.2 with Theorem 6.6:
[Equation]
In turn, Reg ` ( f 0 ) in the RHS of (8) can be upper bounded as in (7).
Conclusions
The results we presented aim at ﬁlling the gap in the statistical analysis of MLC problems and methods.
In the non-parametric setting, we provided upper and lower ﬁnite-sample regret bounds for k -NN classiﬁers in MLC with Hamming loss and Precision@  under suitable smoothness assumptions.
In fact, our analysis can be extended to other loss functions, like rank loss (e.g., [11]) and microand macro-averaged complex performance measures (e.g., [22]).
The crucial point is formulating the right assumptions for those loss functions.
The general steps of these analyses will be similar to those contained in this paper, since the Bayes classiﬁers for these losses are in fact based on marginal quantities [23].
As an other important research avenue to explore is to make our lower bound more tight.
Lower bounds that are given in this paper are tight up to log factors and to weed out this log factor seems challenging in the multi-label setup.
In the parametric setting, we considered general regret reductions from MLC surrogate losses to
Hamming loss and Precision@  , and then we presented regret bounds one can achieve by empirical risk minimization operating on the surrogate losses.
One natural question is to understand to what extent these regret bounds (like those contained in Eqs.
(7) and (8)) are tight, speciﬁcally when the approximation error R ` ( f ⇤ )
R
⇤ ` is strictly positive.