Simplified and Unified Analysis of Various Learning Problems by Reduction to Multiple-Instance Learning
Abstract
In statistical learning, many problem formulations have been proposed so far, such as multi-class learning, complementarily labeled learning, multilabel learning, multi-task learning, which provide theoretical models for various real-world tasks.
Although they have been extensively studied, the relationship among them has not been fully investigated.
In this work, we focus on a particular problem formulation called Multiple-Instance
Learning (MIL), and show that various learning problems including all the problems mentioned above with some of new problems can be reduced to MIL with theoretically guaranteed generalization bounds, where the reductions are established under a new reduction scheme we provide as a byproduct.
The results imply that the MIL-reduction gives a simplified and unified framework for designing and analyzing algorithms for various learning problems.
Moreover, we show that the MILreduction framework can be kernelized.
INTRODUCTION
In this study we explore how a large class of learning problems can be reduced to the Multiple-Instance Learning (MIL) problem.
This is strongly motivated by the results of [Sabato and Tishby, 2012] and [Suehiro et al., 2020].
Suehiro et al.
[2020] showed that some local-feature-based learning problems can be reduced to a MIL problem, which gave us an insight that MIL would have a high capability of representing various learning problems.
Indeed, the reduced problem is too specific whereas Sabato and Tishby [2012] proposed a much more general formulation of MIL and thus we believe that a wider class of learning problems can be reduced to MIL.
We provide a MIL-reduction scheme and reveal that various learning problems, such as multi-class learning, complementarily labeled learning, multi-label learning, and multi-task learning, can be reduced to MIL.
By the reduction, we immediately derive generalization bounds from [Sabato and
Tishby, 2012], as well as learning algorithms.
That is, our reduction scheme greatly simplifies the analyses of generalization bounds as compared with the analyses in the previous works [e.g., Lei et al., 2019, Ishida et al., 2017, Yu et al., 2014, Pontil and Maurer, 2013].
Some of the obtained generalization bounds are competitive or incomparable to the existing results.
In particular, for multi-label learning, we derive an improved generalization bound, and for complementarily labeled learning, we derive a novel learning algorithm, which is the first polynomial-time algorithm in a certain setting.
Moreover, we propose three new learning problems, multi-label learning with perfectionistic loss , top1 ranking learning and top-1 ranking learning with negative feedback , and we demonstrate that they can be reduced to
MIL as well.
The results imply that our MIL-reduction gives a unified scheme for designing and analyzing algorithms for various learning problems.
To provide the MIL-reduction scheme, we propose a general reduction scheme among learning problems.
Our scheme has two remarkable features as described below.
First, our reduction transforms every instance-label pair ( x, y ) in the given sample of the original learning problem to an instancelabel pair ( x ′ , y ′ ) to form a sample of the reduced learning problem.
In contrast, standard reduction schemes employ an instance transformation and an label transformation separately, to construct x ′ from x and y ′ from y , respectively.
Therefore, our scheme enables us to design reduction algorithms among a wider class of learning problems, e.g., learning-to-rank to classification, and supervised learning to weakly supervised learning.
Second, our reduction scheme ensures that the Empirical Risk Minimization (ERM) of the reduced problem implies the ERM of the original one, while the empirical Rademacher complexity of the hypothesis (composed with loss function) classes are preserved through the reduction.
This means that we can employ an existing ERM algorithm for the reduced problem to obtain an ERM algorithm for the original problem with a theoretical guaranteed generalization bound, which is immediately derived from a known generalization bound for the reduced problem.
We also show that the MIL-reduction scheme can be kernelized.
The main contributions are summarized as follows:
• We propose a general reduction scheme based on the ERM, which allows us to derive a generalization risk bound of the original problem immediately.
• We demonstrate that several learning problems, from traditional to new problems, can be reduced to MIL.
The results imply that our MIL-reduction gives a simplified and unified scheme for the analyses for various learning problems.
• We obtain novel theoretical results for some learning problems.
• We show that the MIL-reduction scheme can be kernelized.
Several proofs are shown in supplementary materials.
PRELIMINARIES
For an integer u , [ u ] denotes the set { 1 , . . .
, u } .
I (e) denotes the indicator function of the event e , that is, I (e)
= 1 if e is true
and I (e) = 0 otherwise.
A learning problem is represented by a pair ( H , ℓ ) of a hypothesis class H ⊆ { h : X → Y} and a loss function ℓ : X
×Y×H
→ R for some input space X and output space
Y .
A learner receives a sample S =
(( x 1 , y 1 ) , . . .
, ( x n , y n )) where each input-output pair ( x i , y i ) is drawn i.i.d.
according to an unknown distribution D over X × Y .
The goal of the learner is to find, with high probability, a hypothesis h ∈ H so that the generalization risk R D ( h ) =
E ( x,y ) ∼D ℓ ( x, y, h ) is small.
For a learning problem ( H , ℓ ) , we define a class of loss functions as (cid:98) H
= { ( x, y ) (cid:55)→ ℓ ( x, y, h ) | h ∈ H} when the underlying loss function ℓ is clear from the context.
We give the definition of the empirical Rademacher complexity, which is used to bound the generalization risk.
Definition (Empirical
Rademacher complexity [Bartlett and Mendelson, 2003]) .
Given a sample
S =
(( x 1 , y 1 ) , . . .
, ( x n , y n )) ∈ ( X × Y ) n , the empirical
Rademacher complexity R S ( (cid:98) H ) of a class (cid:98) H w.r.t.
S is defined as R S ( (cid:98) H ) =
1 n E σ (cid:104) sup g ∈ (cid:98) H (cid:80) n i =
σ i g ( x i , y i ) (cid:105) , where σ ∈ {− 1 , 1 } n and each σ i is an independent uniform random variable taking values in {− 1 , +1 } .
Generalization risk bound [Mohri et al., 2018] Let ( H , ℓ ) be a learning problem and S be a sample of size n drawn according to a distribution D .
Then, it holds with probability at least 1 − δ that for all h ∈ H ,
[Equation]
where (cid:98) R S ( h ) = 1 n (cid:80) ni
=1 ℓ ( x i , y i , h ) denotes the empirical risk of h for sample S .
REDUCTION SCHEME FOR ERM
We propose a general reduction scheme for empirical risk minimization and provide useful theoretical results.
Definition 2 (ERM-reduction) .
A learning problem ( H , ℓ ) over input-output space X ×Y is ERM-reducible to another learning problem ( H ′ , ℓ ′ ) over input-output space X ′
× Y ′ if there exist polynomial-time computable functions
α : X ×
Y → X ′ × Y ′ and β : H ′
→ H such that for any ( x, y ) ∈
X × Y and for any h ′ ∈ H ′ , ℓ ( x, y, h ) =
ℓ ′ ( x ′ , y ′ , h ′ ) , where ( x ′ , y ′ )
=
α ( x, y ) and h =
β ( h ′ ) .
Here we show the remarkable relationship between the original problem and the reduced problem.
Proposition 1.
Suppose that ( H , ℓ ) is ERM-reducible to ( H ′ , ℓ ′ ) with transformations α and β .
For any sample S
=
(( x 1 , y 1 ) , . . .
, ( x n , y n )) ∈ ( X × Y ) n , the following holds:
(i) (In)equality of the ERMs:
[Equation]
where H β = { β ( h ′ ) |
h ′ ∈ H ′ } and S ′
=
(( x ′ 1 , y ′ 1 ) , . .
, ( x ′ n , y ′ n )) with ( x ′ i , y ′ i ) =
α ( x i , y i ) for i ∈ [ n ] .
(ii) Empirical Rademacher complexity preserving:
[Equation]
We can design a reduction scheme in a straightforward way as follows.
When given a sample S of the original problem, we construct S ′ of the reduced problem by α and obtain h ′ by solving the ERM of the reduced problem.
Then, we obtain the final hypothesis h by β .
We derive the following generalization risk bound using the propositions on the empirical Rademacher complexity.
Corollary 2.
Let S =
(( x 1 , y 1 ) , . . .
, ( x n , y n )) be a sample i.i.d.
drawn according to unknown distribution D in an original problem ( H , ℓ ) .
If ( H , ℓ ) is ERM-reducible to ( H ′ , ℓ ′ ) , for S ′
= ( α ( x 1 , y 1 ) , . .
, α ( x n , y n )) and
h = β ( h ′ ) , the following generalization risk bound holds with a probability at least 1 − δ for all h ∈ H β :
[Equation]
That is, we can guarantee the generalization bound of the original problem because of the preservation of the empirical
Rademacher complexity.
MIL-REDUCTION FRAMEWORK
This section is the highlight of this paper.
We define the
ERM-reducibility to MIL and show the reducible condition.
Moreover, we show that some theoretical analyses can be simplified.
We use some symbols with prime (e.g., X ′ ) to indicate that the MIL is the reduced problem.
4.1 PROBLEM FORMULATION OF MIL
Let Z ⊆
R
d
′ be the instance space.
X ′ ⊆
2 Z is an input space and a bag x ′ ∈ X ′ is a finite set of instances chosen from Z .
Let Y ′
= {− 1 , 1 } be an output space.
Following the formulation by [Sabato and Tishby, 2012], we define, for the rest of the paper, a MIL problem as a pair ( H ′ , ℓ ′ ) of a hypothesis class H ′ and a loss function ℓ ′ of the form:
[Equation]
where G ⊆ { g : Z → R } , f 1 : R → [0 , 1] is an a -Lipschitz function, f 2 : R → [ − 1 , 1] is a b -Lipschitz function, and Ψ p : 2 [ − 1 , 1] → [ − 1 , 1] is a p -norm like function, which is defined for any p ∈ [1 , ∞ ) as
[Equation]
for every finite set V = { v 1 , v 2 , . .
. , v m } ⊆
[ − 1 , 1] .
We define Ψ ∞ as lim p →∞ Ψ p .
Note that Ψ p is 1 -Lipschitz for any p [see, Sabato and Tishby, 2012].
In MIL tasks, Ψ p is a user-defined function and behaves as an aggregation of some bag information.
Typical Ψ p are the max operator ( p = ∞ ) and average ( p = 1 ).
The only difference in the hypothesis of [Sabato and Tishby, 2012] is f 2 .
f 2 appears redundant (because f 2 ◦ g can be replaced by a single function) but plays an important role in the reduction (the examples are shown in Section 5).
Here we give the definition of ERM-reducibility in a straightforward way.
Definition 3 (MIL-reducibility) .
A learning problem ( H , ℓ ) is said to be MIL-reducible if there exists a MIL problem ( H ′ , ℓ ′ )
such that ( H , ℓ ) is ERM-reducible to ( H ′ , ℓ ′ ) .
Hereinafter, the scheme for ERM-reduction to MIL is called
MIL-reduction scheme .
4.2 RADEMACHER COMPLEXITY BOUND
We show the empirical Rademacher complexity bound for the MIL-reducible problems using our reduction scheme.
As aforementioned, the main advantage of our reduction scheme is to allow us to apply the empirical Rademacher complexity bound of the reduced problem to the original problems.
In this paper, we utilize the bound provided by Sabato and Tishby [2012].
Theorem 3 (An application of Theorem 20 of [Sabato and
Tishby, 2012]) .
Let ( H ′ , ℓ ′ ) be a MIL problem defined in
Eq.
(1) and (2) .
Let S ′
=
(( x ′ 1 , y ′ 1 ) , . .
, ( x ′ n , y ′ n )) be a sample with average bag size r S ′ .
Let (cid:98) G = { f 2 ◦ g | g ∈ G} .
If there exist C, ρ ≥ 0 such that for all sufficiently large n ,
[Equation]
, then
[Equation]
where (cid:98) H ′
= { ˆ h ′ :
x ′ (cid:55)→ f 1 ( y ′ h
( x ′ ))
| h ′ ∈ H ′ } .
As mentioned in [Sabato and Tishby, 2012], we obtain the following bound when G is a set of linear functions.
Corollary 4.
Let G =
{ g : z (cid:55)→ ⟨ w ′ , z ⟩ | w ′ ∈
R d ′ , ∥ w ′ ∥ ≤ C 1 } and assume that ∥ z ∥ ≤ C 2 .
Then, the following bound holds:
[Equation]
The above bound is easily derived from the result of R S ′
[see the proof of Theorem 20 of Sabato and Tishby, 2012]) and R S ′
( (cid:98) G ) ≤ b R S ′ ( G ) ≤ bC 1 C 2 / √ n = bC 1 C 2 ln 0 ( n ) / √ n [see, e.g., Theorem 5.8 and 5.10 of Mohri et al., 2018].
Using Theorem 3 and Corollary 2, we obtain a generalization risk bound for MIL-reducible problems.
4.3 LEARNING ALGORITHM
We show that, under mild conditions, the ERM of MIL becomes a convex or a DC (Difference of Convex) programming problem.
Suppose that G is a set of linear functions:
[Equation] [Equation]
Let S ′
=
(( x ′ 1 , y ′ 1 ) , . .
, ( x ′ n , y ′ n )) .
The ERM of MIL is formulated as follows:
[Equation]
For the optimization problem (4), we show that the following propositions hold.
Proposition 5.
If y ′ i = − 1 for any i ∈ [ n ] for sample S ′ , fis convex and nonincreasing 1 , and f 2 is a nondecreasing convex function, and G is given as Eq.
(3) , then the ERM of ( H ′ , ℓ ′ ) is a convex programming problem.
Proposition 6.
If f 1 is a nonincreasing convex 1 and f 1 ( c ) is a homogeneous function of degree 1 for c ∈ [ − 1 , 1] 2 , and fis a nondecreasing convex function, and G is given as Eq.
(3) , then ERM of ( H ′ , ℓ ′ ) is a DC programming problem.
Generally, it is difficult to find a global minimum for a DC programming problem; however, it is known that we can find a solution with ϵ -approximation of local optima [see, e.g., Le Thi and Dinh, 2018].
We introduce a standard DC algorithm to solve (4) in Algorithm 1 in Sec.
D.
The propositions indicate that, if ( H , ℓ ) is MIL-reducible to ( H ′ , ℓ ′ ) and satisfies either of the above conditions, then the solution h ∈ H β in the original problem can be obtained by a unified learning algorithm.
MIL-REDUCIBLE EXAMPLES
In this section, we demonstrate that various learning problems can be reduced to MIL by the proposed reduction scheme.
The results imply that our MIL-reduction gives a unified scheme for designing and analyzing learning algorithms for various learning problems 3 .
5.1 THE EXISTING PROBLEMS 5.1.1 Multi-class learning problem
Problem setting: Let X ⊆
R
d be an instance space, and
Y =
[ k ] be an output space.
The learner receives the set of labeled instances S =
(( x 1 , y 1 ) , . .
, ( x n , y n )) ∈ ( X × Y ) n , where each instance is drawn i.i.d.
according to some unknown distribution D .
The learner predicts the label of x using the hypothesis h ∈ H
=
{ x (cid:55)→ arg max j ∈ [ k ] ⟨ w j , x ⟩ | ∀ j ∈
[ k ] , w j ∈ R d } .
Let ℓ : ( x, y, h ) (cid:55)→ Γ( ⟨ w y , x ⟩ − max j ∈Y\ y ⟨ w j , x ⟩ ) be a loss function, where Γ : R → [0 , 1] is a convex, nonincreasing and a -Lipschitz function.
The generalization risk and empirical risk of h are defined as:
[Equation]
We obtain the following by using MIL-reduction scheme:
Theorem 7.
Multi-class learning problem is MIL-reducible.
Proof.
For any ( x, y ) , we define
[Equation]
where 0 is a d -dimensional vector, the elements of which are all 0 .
On the MIL-reduction scheme, suppose that p = ∞ ;
[Equation]
[Equation]
c ) [Equation]
[Equation] 2 (shifting function to [ − 1 , +1] ); α ( x, y ) =
( x ′ ( x,y ) , y ′ )
where x ′ ( x,y ) =
{ η ( x,j ) − η ( x,y ) | ∀ j ∈ Y\ y } ; y ′ =
− 1 ; for any z ∈ R kd ,
G = { g : z (cid:55)→ ⟨ ( w ′ 1 , . . . , w ′ k ) , z ⟩ | w ′ j ∈ R d , ∀ j ∈ [ k ] , ∥ W ′ ∥
≤ C 1 } where W ′ =
( w ′ 1 , . . .
, w ′ k ) and ∥ W ′ ∥
= (cid:113)(cid:80) kj =1 ∥
w ′ j ∥ 2 ; β ( h ′ ) :
x (cid:55)→ arg max j ∈ [ k ] ⟨ w ′ j , x ⟩ .
Then, for any ( x, y ) and h ∈ H , ℓ ′ ( x ′ , y ′ , h
[Equation]
(cid:111)(cid:17)(cid:17)(cid:17)
[Equation]
(cid:111)(cid:17)(cid:17)(cid:19)
[Equation]
(cid:111)(cid:17)(cid:17)(cid:19)
[Equation]
(cid:111)(cid:17)(cid:17)(cid:19)
[Equation]
The empirical Rademacher complexity is immediately derived as follows by observing the reduction process.
Corollary 8.
We assume that ∥ x i ∥ ≤ C 2 for any i ∈ [ n ] .
In the reduced MIL problem from multi-class learning problem, the empirical Rademacher complexity of (cid:98) H ′ is given as:
[Equation]
where ˆ a = 2 aC 1 C 2 and we assume ∥ w ′ ∥ ≤ C 1 in the reduced MIL.
We used the fact that the bag size is ( k − 1) for all x ′ i (i.e., r S ′ =
k − 1 ) and R ( (cid:98) G ) ≤ 2 / √ n by setting f 2 ( c ) =
c / C 1 C 2 .
Using Corollary 2, we can obtain the generalization risk bound for the multi-class learning.
The learning algorithm is obtained by the following result.
Corollary 9.
The reduced ERM of the MIL from multi-class learning is a convex programming problem.
The proof of Theorem 7 shows that f 2 is nondecreasing convex and y
′ i = − 1 for all i ∈ [ n ] .
Therefore, by Proposition 5, if we consider Γ that is a nonicreasing and convex function, the ERM of the reduced MIL problem is a convex programming problem and solved in polynomial time.
5.1.2 Complementarily labeled learning problem
Complementarily labeled learning was proposed by Ishida et al.
In this problem, some training instances are complementarily labeled (e.g., instance x i is NOT y i ).
We essentially follow the problem setting and some assumptions provided by Ishida et al.
Problem setting: Let X ⊆
R
d be an instance space and
Y =
[ k ] be an output space.
Let D be an unknown distribution over X × Y .
We assume that the learner receives a sample S drawn i.i.d.
according to the distribution D ′ which provides the true label with unknown probability θ and the complementary label with unknown probability 1 − θ .
Moreover, we assume that the complementary label is chosen with a uniform probability (i.e., all complementary labels are equally chosen with the probability 1 / ( k − 1) ).
More formally, we assume that the sample is given as
S =
(( x 1 , y 1 , γ 1 ) . .
, ( x n , y n , γ n )) which is drawn i.i.d.
according to the distribution D ′ over D × { False , True } , where γ i =
True means that y i is the true label and γ i
=
False means that y i is the complementary label (i.e., it indicates that x i is NOT y i ).
For any ( x, y ) ∼ D ,
D ′
( x, y, True)
= θ and D
′ ( x, ¯ y, False) =
1 − θ k − 1 for any ¯ y ̸
= y (i.e., the complementary label is chosen with a uniform probability).
The other basic settings are the same as those for the aforementioned multi-class learning.
The learner predicts the label of x using the hypothesis h ∈ H
= { x (cid:55)→ arg max j ∈ [ k ] ⟨ w j , x ⟩ | ∀ j ∈ [ k ] , w j ∈ R d } .
The final goal of the learner is to find h ∈ H with a small multi-class classification risk:
[Equation]
However, it is difficult to minimize the empirical multiclass classification risk directly using the complementarily labeled data.
Therefore, we consider the following risk 5 .
[Equation]
This risk implies that when γ =
True , the learner does not incur a risk if it predicts the true label.
When γ =
False , the learner does not incur a risk if it predicts an assigned nontrue label.
Thus, the risk measure is defined using the pair ( y, γ ) ∈
( Y × { False , True } ) .
We can show that achieving a small R LC D
′ ( h ) is consistent with achieving small R MC D ( h ) as follows:
Lemma 1.
For any h ∈ H , R MC D ( h ) =
k
[Equation]
R LC D
[Equation]
( h ) holds.
Thus, minimizing R LC D ′ ( h ) is a reasonable way to achieve a high multi-class classification accuracy.
Generally, there is no loss function ℓ (( x, γ ) , y, h ) which is a convex upper bound on the zero-one loss
I ( γ = ( y ̸ = h ( x ))) over the domain w .
This is because if I ( γ = True) =
1 then max is convex w.r.t.
w ; however, if I ( γ = True) = − 1 then − max = min is concave w.r.t.
w .
Therefore, we consider the convex upper bounded loss only on the risk for complementarily labeled data (i.e., the concave risk for the normally labeled data) using Γ : R → [0 , 1] as Γ (cid:0) max j ∈Y\ y ⟨ ( w j − w y ) , x ⟩ (cid:1) .
We then define the nonconvex risk ℓ ( x, ( γ, y ) , h ) =
Γ (cid:0) I ( γ = True) × (cid:0) max j ∈Y\ y ⟨ ( w j − w y ) , x ⟩ (cid:1)(cid:1) .
The empirical risk is formulated as:
[Equation]
The following is obtained by MIL-reduction scheme.
Theorem 10.
Complementarily labeled learning is MILreducible.
The difference from the reduction in multi-class learning is that only y ′ takes {− 1 , 1 } .
y ′ behaves as a switch that turns the loss of complementarily or normally labeled data.
The empirical Rademacher complexity is bounded as:
Corollary 11.
We assume that ∥ x i ∥ ≤ C 2 for any i ∈ [ n ] .
In the reduced MIL problem from complementarily labeled learning, the empirical Rademacher complexity of (cid:98) H ′ is given by:
[Equation]
where ˆ a = 2 aC 1 C 2 and we assume ∥ w ′ ∥ ≤ C 1 in the reduced MIL problem.
We use the same argument as in Corollary 8.
Using Corollary 2 and Lemma 1, we obtain the generalization bound for the complementarily labeled learning.
The learning algorithm is derived by the following result:
Corollary 12.
The reduced ERM of the MIL from complementarily labeled learning is a DC programming problem.
If the sample contains only complementarily labeled data, the learning problem is a convex programming problem.
Generally, y ′ ∈ {− 1 , 1 } in complementarily labeled learning.
Using the proof of Theorem 10 and by Proposition 6, if we consider Γ( c ) which is a nondecreasing and homogeneous function of degree 1 for c ∈ [ − 1 , 1] such as hingeloss function, we can solve the problem by DC algorithm as shown in Algorithm 1.
Note that, if the sample contains only complementarily labeled data (i.e., ∀ i ∈ [ n ] , y i = − 1 ), it becomes a convex programming problem.
5.1.3 Multi-label learning problem
Problem setting Let X ⊆
R d be an instance space and
Y ∈ {− 1 , 1 } k be an output space, and D be an unknown distribution over X .
Unlike the standard multi-class learning setting introduced in Section 5.1.1, each instance may have multiple labels (e.g., in text-categorization tasks, some texts have multiple topics such as IT and business).
y j denotes the j -th element of y i .
The learner receives a labeled sample S =
( x 1 , y 1 ) , . . .
, ( x n , y n ) ∈
X × Y which is drawn i.i.d.
according to the distribution D .
The learner predicts whether x belongs to class j ∈ [ k ] or not using the hypothesis h ∈ H =
{ ( x, j ) (cid:55)→ sign( ⟨ w j , x ⟩ ) | ∀ w j ∈ R d } .
Let ℓ : ( x, y, h ) (cid:55)→ 1 k (cid:80) kj =1 Γ( − y j ⟨ w j , x ⟩ ) where Γ : R → [0 , 1] is a convex, nondecreasing and b -Lipschitz function 6 .
The generalization and empirical risk of h are defined as:
[Equation]
Reduction to MIL
Theorem 13.
Multi-label learning is MIL-reducible.
Proof.
On the MIL-reduction scheme, suppose that p = 1 ; f 1 : f 1 ( a ) = − a for a ∈ R ; f 2 is Γ ; α ( x, y )
=
( x ′ ( x,y ) , y ′ )
where x ′ ( x,y ) = { ( − y 1 x, 1) , . . .
, ( − y k x, k ) } ; y ′ =
− 1 ; G = { g : ( z, j ) (cid:55)→ ⟨ w ′ j , z ⟩ | w ′ j ∈ R d , ∀ j ∈ [ k ] ,
∥ W ′ ∥ ≤ C 1 } where W ′ =
( w ′ 1 , . . .
, w ′ k ) ; β ( h ′ ) :
( x, j ) (cid:55)→ sign( ⟨ w ′ j , x ⟩ ) .
For any ( x, y ) and h ∈ H , we have that
[Equation]
The empirical Rademacher complexity is bounded as:
Corollary 14.
We assume that ∥ x i ∥ ≤ C 2 for any i ∈ [ n ] .
In the reduced MIL problem, the empirical Rademacher complexity of (cid:98) H ′ is given as follows:
[Equation]
where ∥ w ′ ∥ ≤ C 1 in the reduced MIL.
We used the fact that the size of each bag is k .
Using Corollary 2, we obtain the generalization risk bound for the multilabel learning.
The learning algorithm is obtained by the following result.
Corollary 15.
The reduced ERM of the MIL from multilabel learning is a convex programming problem.
The proof of Theorem 13 shows that, f 1 is nonincreasing and convex, and
y ′ i = − 1 for all i ∈ [ n ] .
Therefore, by
Proposition 5, if we consider Γ that is nondecreasing and convex, the reduced problem is a convex programming problem and it is solved in polynomial time.
5.2 APPLICATION TO THE NEW PROBLEMS 5.2.1 Multi-label learning with perfectionistic loss
Problem setting: In a standard multi-label learning (see
Sec.5.1.3), we consider the average prediction error (loss) with the classes.
On the other hand, we consider a perfectionistic error in multi-label learning problem.
More formally, we consider the following loss in a multi-label learning:
[Equation]
where Γ : R → [0 , 1] is a convex, nondecreasing and b -Lipschitz function.
This loss means that the learner incurs the risk unless the learner perfectly predict the correct labels.
The generalization and empirical risks of h are given as R D
[Equation]
respectively
[Equation]
Using MIL-reduction scheme, we obtain the following:
Theorem 16.
Multi-label learning with perfectionistic loss is MIL-reducible.
This can be derived by the same argument with multi-label learning except for p = ∞ (see Sec.H).
The empirical Rademacher complexity is bounded as:
Corollary 17.
We assume that ∥ x i ∥ ≤ C 2 for any i ∈ [ n ] .
In the reduced MIL problem, the empirical Rademacher complexity of (cid:98) H ′ is given as follows:
[Equation]
where we assume ∥ w ′ ∥ ≤ C 1 .
Interestingly, we can have the same generalization risk bound with the standard multi-label learning.
The learning algorithm is derived by the following result.
Corollary 18.
The reduced ERM of the MIL from multilabel learning with perfectionistic loss is a convex programming problem.
This is easily obtained by observing the reduction process shown in Sec.
H and using Prpoposition 5.
A naive approach for the multi-label learning with perfectionistic loss is to reduce to multi-class learning.
That is, we consider all combinations of the multi-label as multi-classes and solve 2 k -class learning problem with high computational cost.
However, by the above corollary, multi-label learning with perfectionistic loss can be solved efficiently.
5.2.2 Top-1 ranking learning
Learning to rank is a fundamental problem, and many applications, such as recommendation systems, exist.
We consider the following natural scenario in a recommendation problem; a learner has a set that contains several items, and it wishes to recommend an item to a target user from the set.
Problem setting: Let X ⊆
R
d be an instance space, and s : X → R be a target scoring function.
Set A is a finite set of instances selected from X .
The learner receives the sequence of the sets of items and the chosen item
S =
( A 1 , x ∗ 1 ) , . . .
, ( A n , x ∗ n ) , where each x ∗ i ∈
A i is the highest-valued item determined by the target function s .
k denotes the average size of the item sets in S , that is, k = 1 n (cid:80) ni
=
| A i | .
Each sample set of items is drawn i.i.d.
from X according to an unknown distribution D over 2 X .
Assume that the learner predicts the item from the item set using the hypothesis h ∈ H
=
{ A (cid:55)→ arg max x ∈
A ⟨ w, x ⟩ | w ∈ R d } .
7 Let ℓ ( A, x ∗ , h ) is a convex upper bound on the zero-one loss function I ( y ̸
= ˆ y ) .
Equivalently, we consider the zero-one loss I ( ⟨ w, x ∗ ⟩ −
max x ∈
A \ x ∗ ⟨ w, x ⟩ ≤ 0) and its convex upper bounded loss ℓ : ( A, x ∗ , h ) (cid:55)→ Γ( ⟨ w, x ∗ ⟩ − max x ∈
A \ x ∗ ⟨ w, x ⟩ ) where Γ : R → [0 , 1] is a convex, nonincreasing and a Lipschitz function.
The goal of the learner is to find h ∈ H with a small misranking risk w.r.t.
the target s .
Thus, the generalization and empirical risks are formulated as follows:
[Equation]
where x ∗ = arg max x ∈
A s ( x ) .
We obtain the following by using MIL-reduction scheme:
Theorem 19.
Top-1 ranking learning is MIL-reducible.
The reducible condition is satisfied when we set α ( A, x ∗ ) =
( x ′ , y ′ )
where x ′ = { x − x ∗ | x ∈
A \ x ∗ } y ′ i = − 1 for all i ∈ [ n ] .
The details of the reduction process is in Sec.
I.
The empirical Rademacher complexity bound is as follows:
Corollary 20.
We assume that ∥ x ∥ ≤ C 2 for any x ∈
A i , ∀ i ∈ [ n ] .
In the reduced MIL problem, the empirical
Rademacher complexity of (cid:98) H ′ is given as follows:
[Equation]
where ˆ a = 2 aC 1 C 2 and we assume ∥ w ′ ∥ ≤ C 1 .
The generalization bound can be derived by applying r S ′ =
k − 1 and using the fact that ∥ z ∥ ≤ 2 C 2 for any z ∈ x ′
i , ∀ i ∈ [ n ] in the reduced MIL.
By using Corollary 2, we can obtain the generalization risk bound for the Top-1 ranking learning.
The learning algorithm is designed by the following result:
Corollary 21.
The reduced ERM of MIL from top-1 ranking learning is a convex programming problem.
The corollary can be easily derived from the reduction process detailed in I.
Extension: We consider top-1 ranking learning with negative feedback which is an extension of top-1 ranking learning.
We show the details in Sec.
J. Remarkably, the ERM problem of the reduced MIL is a DC programming problem.
KERNELIZED EXTENSION
Although we consider a linear function set as G ; in practice, a nonlinear kernel is required for various learning tasks.
A straightforward method is to employ a kernel-approximation technique [see, e.g., Sec.6.6 in Mohri et al., 2018], which constructs feature vectors Φ( x ) ∈ R D with the theoretical guarantee that ⟨ Φ( x 1 ) , Φ( x 2 ) ⟩ ≈ K ( x 1 , x 2 ) for a userdetermined dimension D .
However, we can use only a limited number of kernels via the approximation technique.
Therefore, we show the kernelized version of the reduction.
6.1 SETTINGS
We assume that an original problem is defined by H , ℓ, X , Y , and Φ : X → H , where H is a reproducing kernel Hilbert space associated to K ( x 1 , x 2 ) = ⟨ Φ( x 1 ) , Φ( x 2 ) ⟩ .
Aside from the computability, we can virtually consider the sample as S =
((Φ( x 1 ) , y 1 ) , . . .
, (Φ( x n ) , y n )) .
The ERMreducible condition is that there exist ( x ′ , y ′ )
=
α (Φ( x ) , y ) , h = β ( h ′ ) and ℓ ′ that satisfies ℓ (Φ( x ) , y, h ) =
ℓ ′ ( x ′ i , y ′
i , h ′ )
for any
( x
y
∈ X
×
Y
Let S ′
=
(( x ′ 1 , y ′ 1 ) , . .
, ( x ′ n , y ′ n )) and let G = { g : z (cid:55)→ ⟨ w ′ , z ⟩ | w ′ ∈ H ′ } .
We assume that ( H , ℓ ) is MIL-reducible to H ′ , ℓ ′ .
The ERM of the reduced MIL is formulated as:
[Equation]
where L w ′
=
(cid:80) ni f
( y ′ i Ψ p ( { f 2 ( ⟨ w ′ , z ⟩ | z ∈ x ′ i ) } )) .
6.2 COMPUTABILITY
We show that the representer theorem holds for the optimization problem (5).
Theorem 22 (Representer theorem) .
An optimal solution of the ERM problem (5) has the form ˜ w ′
=
(cid:80) z ∈ P S ′ µ z z , where P S ′
= (cid:83) ni
=1 x ′ i .
Thus, the ERM problem (5) is equivalently formulated as:
[Equation]
µ
[Equation] [Equation]
+ L µ , where L µ = (cid:80) ni
=1 f
( y i Ψ p ( { f 2 ( (cid:80)
z ∈ P S ′ µ z ⟨ z, ˆ z ⟩ ) | ˆ z ∈ x ′ i } )) .
Therefore, if ⟨ z 1 , z 2 ⟩ is polynomial-time computable for any z 1 , z 2 ∈ x ′ using the original kernel function K as an oracle, the ERM of the MIL can be solved similar to linear case according to the condition in Proposition 5 and 6 (DC algorithm for the kernel version is in Sec.
L).
For all
MIL-reducible problems introduced in the paper, ⟨ z 1 , z 2 ⟩ is polynomial-time computable using K (see details in Sec.
M).
Moreover, we can construct β in polynomial time.
DISCUSSION 7.1 RELATED WORK
Other reduction techniques: Several machine-learning reduction schemes exist [see, e.g., Beygelzimer et al., 2015], and we found general reduction schemes, such as [Pitt and
Warmuth, 1990, Beygelzimer et al., 2005].
A major difference between the proposed scheme and existing approaches is that we focus on the reduction of ERM.
Various applications of machine learning reductions, such as reduction from multi-class learning to binary classification
[James and
Hastie, 1998, Ramaswamy et al., 2014], and from ranking to binary classification [Balcan et al., 2008, Ailon and Mohri, 2010, Agarwal, 2014], exist.
To the best of our knowledge, the reduction to MIL has not yet been discussed.
Multi-Class Learning:
Recently, Lei et al.
[2019] achieved log( k ) -dependent generalization bound.
The proposed generalization bound is competitive with the bound.
However, our derivation is highly simpler than the analysis of [Lei et al., 2019] because the reduction allows us to apply the existing MIL bound of [Sabato and Tishby, 2012].
Complementarily-labeled learning: Ishida et al.
[2017] provided the generalization risk bound in the case in which the training sample contains only complementarily labeled instances (i.e., θ = 0 ).
The proposed generalization bound is incomparable to the bound (see details in Sec.N).
Ishida et al.
[2017] selected nonconvex loss functions and optimized the empirical risks using a gradient-based algorithm in practice.
However, there is no guarantee of the optimality of the solution.
We show that the learning problem can be solved by DC algorithm and guarantee the local optima.
Moreover, in the special case that sample contains only complementarily labeled data, the learning problem becomes convex programming and we can obtain global optima.
To the best of our knowledge, the provided learning algorithm is a first polynomial-time algorithm in the special case.
Multi-label learning: Various approaches and generalization analyses have been provided [Yu et al., 2014, Bhatia et al., 2015, Xu et al., 2016a,b].
However, to the best of our knowledge, this paper is the first to propose a log( k ) - dependent generalization bound for the linear (or nonlinear kernel) hypothesis class, where k is the number of classes.
Multi-task learning: A similar generalization bound was reported by [Pontil and Maurer, 2013].
Their results suggest the advantage of regularizing the weights w 1 , . . .
, w T over
T tasks.
However, our result is derived from an entirely different argument from [Pontil and Maurer, 2013] and the derivation is highly simplified.
Top-1 ranking learning: Top-1 ranking measure was originally discussed in [Hidasi and Karatzoglou, 2018].
However, the basic problem setting is different from ours.
They assumed that the recommender has i.i.d.
positive and negative items as the sample.
Moreover, they did not propose a general form of the problem and theoretical analysis.
MIL: MIL was originally proposed by Dietterich et al.
[1997], which is known as weakly supervised learning and there have been proposed many real applications [Gärtner et al., 2002, Andrews et al., 2003, Zhang et al., 2013, Doran and Ray, 2014, Carbonneau et al., 2018].
The generalization bound and learning algorithm have been analyzed from the theoretical perspective [Sabato and Tishby, 2012, Doran, 2015, Suehiro et al., 2020].
There have been several studies on the relationship between MIL with other learning tasks.
Zhou and Xu [2007] showed that a classical MIL can be considered as specific semi-supervised learning.
Zhang et al.
[2020] utilized MIL for extracting causal instances.
However, these works do not imply any type of reduction in the sense of computation theory: if problem A is reduced to B, then we should immediately obtain an algorithm for
A from any algorithm for B combined with the reduction (input-output transformations) with a certain performance guarantee.
Suehiro et al.
[2020] found that a local-feature- based time-series classification problem can be reduced to a MIL problem with a generalization risk bound.
However, the reduced problem is too specific.
Our results first show that various learning problems can be reduced to MIL.
7.2 PRACTICAL IMPLICATIONS
An important contribution of the paper in both the theoretical and practical aspects is to provide a simple and general reduction scheme among various learning problems with theoretical guarantees on generalization bounds.
This means that when faced with a new learning problem A, we can search for an existing ERM problem B that is reducible from A. If succeeded, then we immediately obtain a learning algorithm for A with a generalization bound.
Usually, this process is expected to be much easier than designing a learning algorithm from scratch.
In particular, we demonstrate that various learning problems are reducible to a particular problem, MIL.
That is, we only have to improve ERM algorithms for MIL, which work on the original learning problems as well.
Moreover, we show that ERM for MIL can be formulated as DC programming problems in Section 4.3.
Therefore, we can employ a state-of-the-art DC programming package, which is rapidly evolving these days [Le Thi and Dinh, 2018].
For instance, complementarily labeled learning, which is only known to have a non-convex optimization formulation [Ishida et al., 2017, 2019], would enjoy the benefits from a promising DC programming approach.
Experiments: We demonstrate that our theoretical results are practically useful in the following experiment on complementarily labeled learning tasks 8 .
We use three artificial datasets and four benchmark datasets available in UCI machine learning repository 9 .
The details of artificial datasets are described in Section O.
For all datasets, all training instances are complementarily labeled uniformly at random.
That is, the ERM problem which is derived from our
MIL-reduction scheme becomes a convex programming problem (quadratic programming problem).
On the other hand, [Ishida et al., 2017] solves a nonconvex optimization problem by using Adam [Kingma and Ba, 2014].
The size of training sample is fixed to 1000 and we used the remaining data as a test set.
Although we did not tune the optimization hyperparameters of [Ishida et al., 2017] (the number of epochs is 200 and the learning rate is 0.01), we stopped the learning at the epoch when the test accuracy was the maximum.
The loss of [Ishida et al., 2017] was fixed to PC loss which was the best-performed loss [see Ishida et al., 2017].
Our regularization parameter is chosen from { 0 . 01 , 1 , 100 } and the regularization parameter of [Ishida
[Table]
et al., 2017] is chosen from { 0 . 01 , 1 , 100 } .
We evaluated the average accuracy over 10 trials.
Table 1 shows that our method achieved higher classification accuracy than [Ishida et al., 2017] on many datasets.
This result indicates that our MIL-reduction scenario for complementarily labeled learning, which is derived from the proposed MIL-reduction scheme, is useful in practice.
Moreover, our ERM algorithm does not require any hyperparameters for the optimization because the optimization problem is a convex programming problem (or DC programming problem when the training sample contains both labeled and complementarily labeled instances).
On the other hand, the learning algorithm provided by Ishida et al.
[2017] solves a nonconvex optimization problem and usually requires several hyperparameters (e.g., learning rate and the number of epochs) of the nonconvex-optimization solver.
7.3 CONCLUSION AND FUTURE WORK
We revealed that various learning problems can be reduced to a MIL problem by our ERM-based reduction scheme.
The results imply that our MIL-reduction gives a simplified and unified scheme for the analyses for various learning problems.
Moreover, we obtained novel theoretical results for some learning problems.
A practical concern is that the applicable loss functions are limited in the current scheme.
For example, some loss functions without satisfying the conditions of MIL-reducibility (e.g., square loss) cannot be used.
We explore the relaxation of the ERM-reducible condition.
An interesting open problem is how the class of
MIL-reducible problems is characterized.
Our results imply that MIL is one of the hardest problems in a certain class
C of learning problems.
In other words, we could say that
MIL is a C-complete problem.
We would like to investigate how the class C is characterized.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI (Grant Number JP19H04067 and JP20H05967) and JST, ACT-X (Grant
Number JPMJAX200G).