Original: Boosting as Frank-Wolfe
Revised: Boosting as Frank-Wolfe

Original: author names withheld

Original: Under Review for OPT

Original: Abstract
Revised: Abstract

Original: Some boosting algorithms, such as LPBoost, ERLPBoost, and C-ERLPBoost, aim to solve the soft margin optimization problem with the ℓ 1 -norm regularization.
Revised: Some boosting algorithms, such as LPBoost, ERLPBoost, and C-ERLPBoost, aim to solve the soft margin optimization problem with the ℓ 1 -norm regularization.

Original: LPBoost rapidly converges to an ϵ -approximate solution in practice, but it is known to take Ω( m ) iterations in the worst case, where m is the sample size.
Revised: LPBoost rapidly converges to an ϵ -approximate solution in practice, but it is known to take Ω( m ) iterations in the worst case, where m is the sample size.

Original: On the other hand, ERLPBoost and C-ERLPBoost are guaranteed to converge to an ϵ -approximate solution in O ( 1 ϵ 2 ln mν ) iterations.
Revised: On the other hand, ERLPBoost and C-ERLPBoost are guaranteed to converge to an ϵ -approximate solution in O ( 1 ϵ 2 ln mν ) iterations.

Original: However, the computation per iteration is very high compared to LPBoost.
Revised: However, the computation per iteration is very high compared to LPBoost.

Original: To address this issue, we propose a generic boosting scheme that combines the FrankWolfe algorithm and any secondary algorithm and switches one to the other iteratively.
Revised: To address this issue, we propose a generic boosting scheme that combines the FrankWolfe algorithm and any secondary algorithm and switches one to the other iteratively.

Original: We show that the scheme retains the same convergence guarantee as ERLPBoost and
Revised: We show that the scheme retains the same convergence guarantee as ERLPBoost and

Original: C-ERLPBoost.
Revised: C-ERLPBoost.

Original: One can incorporate any secondary algorithm to improve in practice.
Revised: One can incorporate any secondary algorithm to improve in practice.

Original: This scheme comes from a unified view of boosting algorithms for soft margin optimization.
Revised: This scheme comes from a unified view of boosting algorithms for soft margin optimization.

Original: More specifically, we show that LPBoost, ERLPBoost, and C-ERLPBoost are instances of the Frank-Wolfe algorithm.
Revised: More specifically, we show that LPBoost, ERLPBoost, and C-ERLPBoost are instances of the Frank-Wolfe algorithm.

Original: In experiments on real datasets, one of the instances of our scheme exploits the better updates of the secondary algorithm and performs comparably with LPBoost.
Revised: In experiments on real datasets, one of the instances of our scheme exploits the better updates of the secondary algorithm and performs comparably with LPBoost.

Original: 1. Introduction
Revised: 1. Introduction

Original: The ℓ 1 -norm regularized soft margin optimization problem, defined later, is a formulation of finding sparse large-margin classifiers based on the linear program (LP).
Revised: The ℓ 1 -norm regularized soft margin optimization problem, defined later, is a formulation of finding sparse large-margin classifiers based on the linear program (LP).

Original: This problem aims to optimize the ℓ 1 -margin by combining multiple hypotheses from some hypothesis class H .
Revised: This problem aims to optimize the ℓ 1 -margin by combining multiple hypotheses from some hypothesis class H .

Original: The resulting classifier tends to be sparse, so ℓ 1 -margin optimization is helpful for feature selection tasks.
Revised: The resulting classifier tends to be sparse, so ℓ 1 -margin optimization is helpful for feature selection tasks.

Original: Off-the-shelf LP solvers can solve the problem, but they are still not efficient enough for a huge class H .
Revised: Off-the-shelf LP solvers can solve the problem, but they are still not efficient enough for a huge class H .

Original: Boosting is a framework for solving the ℓ 1 -norm regularized margin optimization even though H is infinitely large.
Revised: Boosting is a framework for solving the ℓ 1 -norm regularized margin optimization even though H is infinitely large.

Original: Various boosting algorithms have been invented.
Revised: Various boosting algorithms have been invented.

Original: LPBoost [2] is a practical algorithm and often works efficiently in practice.
Revised: LPBoost [2] is a practical algorithm and often works efficiently in practice.

Original: Although LPBoost terminates rapidly, It is shown that it takes Ω( m ) iterations in the worst case„ where m is the number of training examples [8].
Revised: Although LPBoost terminates rapidly, It is shown that it takes Ω( m ) iterations in the worst case, where m is the number of training examples [8].

Original: Shalev-Shwartz and Singer [7] invented an algorithm called Corrective ERLPBoost (we call this algorithm C-ERLPBoost for shorthand) in the paper on ERLPBoost [9].
Revised: Shalev-Shwartz and Singer [7] invented an algorithm called Corrective ERLPBoost (we call this algorithm C-ERLPBoost for shorthand) in the paper on ERLPBoost [9].

Original: C-ERLPBooost and ERLPBoost find ϵ -approximate solutions in O (ln( m/ν ) /ϵ 2 ) iterations, where ν ∈ [1 , m ] is the soft margin parameter.
Revised: C-ERLPBooost and ERLPBoost find ϵ -approximate solutions in O (ln( m/ν ) /ϵ 2 ) iterations, where ν ∈ [1 , m ] is the soft margin parameter.

Original: The difference is the time complexity per iteration; ERLPBoost solves a convex program (CP) for eachiteration, while C-ERLPBooost solves a sorting-like problem. Although ERLPBoost takes much time per iteration, it takes fewer iterations than C-ERLPBoost in practical applications. © .
Revised: The difference is the time complexity per iteration; ERLPBoost solves a convex program (CP) for each

Original: [Table]
Revised: [Table]

Original: One of our work
Revised: One of our work

Revised: Iter.

Revised: bound

Original: [Table]
Revised: [Table]

Original: Problem per iter.
Revised: Problem per iter.

Original: [Table]
Revised: [Table]

Revised: iteration, while C-ERLPBooost solves a sorting-like problem.

Revised: Although ERLPBoost takes much time per iteration, it takes fewer iterations than C-ERLPBoost in practical applications.

Original: For this reason, ERLPBoost is faster than C-ERLPBoost.
Revised: For this reason, ERLPBoost is faster than C-ERLPBoost.

Original: Our primary motivation is to investigate boosting algorithms with provable iteration bounds which perform as fast as
Revised: Our primary motivation is to investigate boosting algorithms with provable iteration bounds which perform as fast as

Original: LPBoost.
Revised: LPBoost.

Original: This paper has two contributions.
Revised: This paper has two contributions.

Original: Our first contribution is to give a unified view of boosting for soft margin optimization.
Revised: Our first contribution is to give a unified view of boosting for soft margin optimization.

Original: We show that LPBoost, ERLPBoost, and CERLPBoost are instances of the Frank-Wolfe algorithm.
Revised: We show that LPBoost, ERLPBoost, and CERLPBoost are instances of the Frank-Wolfe algorithm.

Original: Our second contribution is to propose a generic scheme for boosting based on the unified view.
Revised: Our second contribution is to propose a generic scheme for boosting based on the unified view.

Original: Our scheme combines a standard Frank-Wolfe algorithm and any algorithm and switch one to the other at each iteration in a non-trivial way.
Revised: Our scheme combines a standard Frank-Wolfe algorithm and any algorithm and switch one to the other at each iteration in a non-trivial way.

Original: We show that this scheme guarantees the same convergence rate, O (ln( m/ν ) /ϵ 2 ), as ERLPBoost and C-ERLPBoost.
Revised: We show that this scheme guarantees the same convergence rate, O (ln( m/ν ) /ϵ 2 ), as ERLPBoost and C-ERLPBoost.

Original: One can incorporate any update rule to this scheme without losing the convergence guarantee, so that it take advantage of better updates of the second algorithm in practice.
Revised: One can incorporate any update rule to this scheme without losing the convergence guarantee, so that it take advantage of better updates of the second algorithm in practice.

Original: In particular, we propose to choose LPBoost as the second algorithm to combine and we call the resulting algorithm Modified LPBoost (MLPBoost).
Revised: In particular, we propose to choose LPBoost as the second algorithm to combine and we call the resulting algorithm Modified LPBoost (MLPBoost).

Original: In experiments on real datasets, MLPBoost works comparably with LPBoost and it is
Revised: In experiments on real datasets, MLPBoost works comparably with LPBoost and it is

Original: MLPBoost is the fastest among theoretically guaranteed algorithms as expected.
Revised: MLPBoost is the fastest among theoretically guaranteed algorithms as expected.

Original: 2. Basic definitions
Revised: 2. Basic definitions

Original: This paper considers binary classification boosting.
Revised: This paper considers binary classification boosting.

Original: We use the same notations as in [7].
Revised: We use the same notations as in [7].

Original: Let
Revised: Let

Original: S :
Revised: S :

Original: [Equation]
Revised: = (( x i , y i )) m i =1 ∈

Original:  ) m be a sequence of m -examples, where X is some set.
Revised: ( X × {± 1 } ) m be a sequence of m -examples, where X is some set.

Original: Let
Revised: Let

Original: H ⊂ [ − 1 , +1]
Revised: H ⊂ [ − 1 , +1]

Original: X be a set of hypotheses.
Revised: X be a set of hypotheses.

Original: For simplicity, we assume |H| =
Revised: For simplicity, we assume |H| =

Original: |{ h 1 , h 2 , . . .
Revised: |{ h 1 , h 2 , . . .

Original: , h n }| = n .
Revised: , h n }| = n .

Original: It is convenient to regard each h j ∈ H as a canonical basis vector e j ∈ R n .
Revised: It is convenient to regard each h j ∈ H as a canonical basis vector e j ∈ R n .

Original: Let
Revised: Let

Original: A =
Revised: A =

Original: ( y i h
Revised: ( y i h

Original: j ( x i ))i,j ∈ [ − 1 , +1] m × n be a matrix of size m × n .
Revised: j ( x i )) ∈ [ − 1 , +1] m × n be a matrix of size m × n .

Original: We denote m -dimensional capped probability simplex as P mν :=
Revised: We denote m -dimensional capped probability simplex as P mν :=

Original: { d ∈ [0 , 1 /ν ] m |
Revised: { d ∈ [0 , 1 /ν ] m |

Original: ∥ d ∥ 1 = 1 } , where ν ∈ [1 , m ].
Revised: ∥ d ∥ 1 = 1 } , where ν ∈ [1 , m ].

Original: We write
Revised: We write

Original: P m =
Revised: P m =

Original: P m 1 for shorthand.
Revised: P m 1 for shorthand.

Original: For a set C ⊂ R n , we denote the convex hull of C as CH( C ) :
Revised: For a set C ⊂ R n , we denote the convex hull of C as CH( C ) :

Original: = { P k w k s k | P k w k = 1 , w k ≥ 0 , { s k } k ⊂ C} .
Revised: = { P k w k s k | P k w k = 1 , w k ≥ 0 , { s k } k ⊂ C} .

Original: We say that a function f : R m → R is η -smooth over a convex set C ⊂
Revised: We say that a function f : R m → R is η -smooth over a convex set C ⊂

Original: R m w.r.t.
Revised: R m w.r.t.

Original: a norm ∥ · ∥ if f ( y ) ≤ f ( x ) + ( y − x ) ⊤ ∇ f ( x ) + η 2 ∥ y − x ∥for all x , y ∈ C .
Revised: a norm ∥ · ∥ if f ( y ) ≤ f ( x ) + ( y − x ) ⊤ ∇ f ( x ) + η 2 ∥ y − x ∥for all x , y ∈ C .

Original: The Fenchel conjugate f ⋆ of a function f : R m → [ −∞ , + ∞ ] is defined as f ⋆ ( θ ) = sup d ∈ R m (cid:16)
Revised: The Fenchel conjugate f ⋆ of a function f : R m → [ −∞ , + ∞ ] is defined as f ⋆ ( θ ) = sup d ∈ R m (cid:16)

Original: d ⊤ θ − f ( d ) (cid:17) .
Revised: d ⊤ θ − f ( d ) (cid:17) .

Original: It is well known that if f is a 1 /η -strongly convex function w.r.t.
Revised: It is well known that if f is a 1 /η -strongly convex function w.r.t.

Original: the norm ∥ · ∥ for some η > 0, f ⋆ is an η -smooth function w.r.t.
Revised: the norm ∥ · ∥ for some η > 0, f ⋆ is an η -smooth function w.r.t.

Original: the dual norm ∥ · ∥ ⋆ .
Revised: the dual norm ∥ · ∥ ⋆ .

Original: Further, if f is a strongly convex function, the gradient vector of f ⋆ is written as ∇ f ⋆ ( θ ) =
Revised: Further, if f is a strongly convex function, the gradient vector of f ⋆ is written as ∇ f ⋆ ( θ ) =

Original: arg sup d ∈ R m (cid:16)
Revised: arg sup d ∈ R m (cid:16)

Original: d ⊤ θ − f ( d ) (cid:17) .
Revised: d ⊤ θ − f ( d ) (cid:17) .

Original: You can find the proof of these properties here [1, 7].
Revised: You can find the proof of these properties here [1, 7].

Original: We define the soft margin optimization problem as the dual problem of the edge minimization problem.
Revised: We define the soft margin optimization problem as the dual problem of the edge minimization problem.

Original: The edge minimization problem is defined as γ :=
Revised: The edge minimization problem is defined as γ :=

Original: min d max j ∈ [ n ] ( d ⊤ A ) j + f ( d ), where f : R m → { 0 , + ∞} is the barrier function such that f ( d ) =
Revised: min d max j ∈ [ n ] ( d ⊤ A ) j + f ( d ), where f : R m → { 0 , + ∞} is the barrier function such that f ( d ) =

Original: ⇐⇒ d ∈ P mν .
Revised: ⇐⇒ d ∈ P mν .

Original: The quantity ( d ⊤ A ) j = P mi =1 d i y i h
Revised: The quantity ( d ⊤ A ) j = P mi =1 d i y i h

Original: j
Revised: j

Original: ( x i ) is often called the edge of the hypothesis
Revised: ( x i ) is often called the edge of the hypothesis

Original: h j w.r.t.
Revised: h j w.r.t.

Original: the distribution d .
Revised: the distribution d .

Original: [7] proved that the ℓ 1 -norm regularized soft margin optimization problem is derived as the Fenchel dual problem of edge minimization: ρ :=
Revised: [7] proved that the ℓ 1 -norm regularized soft margin optimization problem is derived as the Fenchel dual problem of edge minimization: ρ :=

Original: max w ∈P n − f ⋆ ( − A w ) = max w ∈P n min d ∈P mν d ⊤ A w .
Revised: max w ∈P n − f ⋆ ( − A w ) = max w ∈P n min d ∈P mν d ⊤ A w .

Original: Since the strong duality holds, γ = ρ .
Revised: Since the strong duality holds, γ = ρ .

Original: The goal of the soft margin optimization is to find an optimal combined hypothesis H T
Revised: The goal of the soft margin optimization is to find an optimal combined hypothesis H T

Original: = P nj =

Revised: = P nj =

Original: w ⋆j h j , where w ⋆ ∈ P n is an optimal solution of the soft margin optimization.
Revised: w ⋆j h j , where w ⋆ ∈ P n is an optimal solution of the soft margin optimization.

Original: Although the edge minimization and soft margin optimization problems are formulated as a linear program, solving the problem for a huge class H is hard.
Revised: Although the edge minimization and soft margin optimization problems are formulated as a linear program, solving the problem for a huge class H is hard.

Original: Boosting is a standard approach to dealing with the problem.
Revised: Boosting is a standard approach to dealing with the problem.

Original: Boosting is formulated by a protocol between two algorithms; the booster and the weak learner.
Revised: Boosting is formulated by a protocol between two algorithms; the booster and the weak learner.

Original: For each iteration t = 0 , 1 , 2 , . . .
Revised: For each iteration t = 0 , 1 , 2 , . . .

Original: , T , the booster chooses a distribution d t ∈ P mν over the training examples S .
Revised: , T , the booster chooses a distribution d t ∈ P mν over the training examples S .

Original: Then, the weak learner returns a hypothesis h
Revised: Then, the weak learner returns a hypothesis h

Original: j t +1 ∈ H to the booster that satisfies ( d t ⊤ A )
Revised: j t +1 ∈ H to the booster that satisfies ( d t ⊤ A )

Original: j t +1 ≥ g for some unknown guarantee g > 0.
Revised: j t +1 ≥ g for some unknown guarantee g > 0.

Original: The boosting algorithm aims to produce a convex combination H T = P
Revised: The boosting algorithm aims to produce a convex combination H T = P

Original: Tt =1 w T,t h
Revised: Tt =1 w T,t h

Original: j t of the hypotheses { h j 1 , h j 2 , . . .
Revised: j t of the hypotheses { h j 1 , h j 2 , . . .

Original: , h j T } ⊂ H that satisfies
Revised: , h j T } ⊂ H that satisfies

Original: [Equation]
Revised: [Equation]

Original: 1) for any predefined ϵ > 0.
Revised: for any predefined ϵ > 0.

Original: Suppose that the weak learner always returns a max-edge hypothesis.
Revised: Suppose that the weak learner always returns a max-edge hypothesis.

Original: In that case, the goal is to find an ϵ -approximate solution of the soft margin optimization.
Revised: In that case, the goal is to find an ϵ -approximate solution of the soft margin optimization.

Original: 3. Main results
Revised: 3. Main results

Original: We first show the unified view of the boosting algorithms via Fenchel duality.
Revised: We first show the unified view of the boosting algorithms via Fenchel duality.

Original: From this view, LPBoost, ERLPBoost, and C-ERLPBoost can be seen as instances of the Frank-Wolfe algorithm with different step sizes and objectives.
Revised: From this view, LPBoost, ERLPBoost, and C-ERLPBoost can be seen as instances of the Frank-Wolfe algorithm with different step sizes and objectives.

Original: Using this knowledge, we derive a new boosting scheme.
Revised: Using this knowledge, we derive a new boosting scheme.

Original: Note that all proofs are found in appendix.
Revised: Note that all proofs are found in appendix.

Original: 3.1. A unified view of boosting for soft margin optimization
Revised: 3.1. A unified view of boosting for soft margin optimization

Original: This section assumes that the weak learner always returns a hypothesis h ∈ H that maximizes the edge w.r.t.
Revised: This section assumes that the weak learner always returns a hypothesis h ∈ H that maximizes the edge w.r.t.

Original: the given distribution.
Revised: the given distribution.

Original: We start by revisiting C-ERLPBoost.
Revised: We start by revisiting C-ERLPBoost.

Original: C-ERLPBoost aims to solve the convex program min
Revised: C-ERLPBoost aims to solve the convex program min

Original: d max j ∈ [ n ] ( d ⊤ A ) j + ˜ f
Revised: d max j ∈ [ n ] ( d ⊤ A ) j + ˜ f

Original: ⋆ ( d ) , , where ˜ f
Revised: ⋆ ( d ) , , where ˜ f

Original: = f + 1 η ∆.
Revised: = f + 1 η ∆.

Original: Since 1 η ∆ is a 1 η -strongly convex function w.r.t.
Revised: Since 1 η ∆ is a 1 η -strongly convex function w.r.t.

Original: ℓ 1 -norm, so does ˜ f .
Revised: ℓ 1 -norm, so does ˜ f .

Original: One can easily verify that the dual problem is max w ∈P n − ˜ f
Revised: One can easily verify that the dual problem is max w ∈P n − ˜ f

Original: ⋆ ( − A w ).
Revised: ⋆ ( − A w ).

Original: Furthermore, ˜ f
Revised: Furthermore, ˜ f

Original: ⋆ is an η -smooth function w.r.t.
Revised: ⋆ is an η -smooth function w.r.t.

Original: ℓ ∞ -norm.
Revised: ℓ ∞ -norm.

Original: Thus, the soft margin optimization problem becomes a minimization problem of a smooth function.
Revised: Thus, the soft margin optimization problem becomes a minimization problem of a smooth function.

Original: C-ERLPBoost updates the distributiond t ∈ P mν as d t = ∇ ˜
Revised: C -ERLPBoost updates the distribution

Original: f ⋆ ( θ t ) = arg min d d ⊤ θ t + ˜ f
Revised: [Equation]

Original: ( d ), where θ t = − A w t .
Revised: where θ t = − A w t .

Original: Then, obtain a hypothesis h
Revised: Then, obtain a hypothesis h

Original: j t +1 ∈ H that maximizes the edge; j t +1 ∈arg max j ∈ [ n ] ( d ⊤ t A ) j . We can rewrite this step as [Equation]
Revised: j t +1 ∈ H that maximizes the edge; j t +1 ∈

Original: Input: A matrix A
Revised: Input: A matrix A

Original: =
Revised: =

Original: ( y i h
Revised: ( y i h

Original: j ( x i ))
Revised: j ( x i ))

Original: i,j ∈ [ − 1 , 1] m × n , a Frank-Wolfe rule F , a secondary algorithm B , and parameters ν > 0 and ϵ > 0.
Revised: i,j ∈ [ − 1 , 1] m × n , a Frank-Wolfe rule F , a secondary algorithm B , and parameters ν > 0 and ϵ > 0.

Original: 1: Set w 0 = 0 .
Revised: 1: Set w 0 = 0 .

Original: 2: for t = 0 , 1 , 2 , . . .
Revised: 2: for t = 0 , 1 , 2 , . . .

Original: , T do 3:
Revised: , T do 3:

Original: Compute the distribution d t = ∇ ˜
Revised: Compute the distribution d t = ∇ ˜

Original: f ⋆ ( − A w t ).
Revised: f ⋆ ( − A w t ).

Original: Obtain a hypothesis h
Revised: Obtain a hypothesis h

Original: j t +1 ∈ H and set E t
Revised: j t +1 ∈ H and set E t

Original: +1 = { e j τ } t +1 τ =1 .
Revised: +1 = { e j τ } t +1 τ =1 .

Original: 5: if ϵ t := min 0 ≤ τ ≤ t ( d ⊤ τ A ) j τ +1 + ˜ f
Revised: 5: if ϵ t := min 0 ≤ τ ≤ t ( d ⊤ τ A ) j τ +1 + ˜ f

Original: ⋆ ( − A w t ) ≤ ϵ/ 2 then Set T = t , break .
Revised: ⋆ ( − A w t ) ≤ ϵ/ 2 then Set T = t , break .

Original: Compute the Frank-Wolfe weight w (1) t +1 =
Revised: Compute the Frank-Wolfe weight w (1) t +1 =

Original: F ( A, w t , e j t +1 , E t , d t ).
Revised: F ( A, w t , e j t +1 , E t , d t ).

Original: Compute the secondary weight w (2) t +1 =
Revised: Compute the secondary weight w (2) t +1 =

Original: B ( A, E t +1 ).
Revised: B ( A, E t +1 ).

Original: Update the weight w t +1 ← arg min w ( k ) t +1 : k ∈{ 1 , 2 } ˜ f
Revised: Update the weight w t +1 ← arg min w ( k ) t +1 : k ∈{ 1 , 2 } ˜ f

Original: ⋆ ( − A w ( k ) t +1 ).
Revised: ⋆ ( − A w ( k ) t +1 ).

Original: 9: end for
Revised: 9: end for

Original: Output: Combined classifier H T =
Revised: Output: Combined classifier H T =

Original: P T t =1 w T,t h t .
Revised: P T t =1 w T,t h t .

Revised: arg max j ∈ [ n ] ( d ⊤ t A ) j .

Revised: We can rewrite this step as

Original:  Thus, finding a hypothesis that maximizes edge corresponds to solving linear programming in the Frank-Wolfe algorithm.
Revised: [Equation] Thus, finding a hypothesis that maximizes edge corresponds to solving linear programming in the Frank-Wolfe algorithm.

Original: Further, C-ERLPBoost updates the weights as w t +1 =
Revised: Further, C-ERLPBoost updates the weights as w t +1 =

Original: w t + λ t ( e j t +1 − w t ), where λ t is the short-step, as in algorithm 3.
Revised: w t + λ t ( e j t +1 − w t ), where λ t is the short-step, as in algorithm 3.

Original: From these observations, we can say that the C-ERLPBoost is an instance of the Frank-Wolfe algorithm.
Revised: From these observations, we can say that the C-ERLPBoost is an instance of the Frank-Wolfe algorithm.

Original: Since ˜ f
Revised: Since ˜ f

Original: ⋆ is η -smooth, we can say that C-ERLPBoost converges in O ( η/ϵ ) =
Revised: ⋆ is η -smooth, we can say that C-ERLPBoost converges in O ( η/ϵ ) =

Original: O (ln( m/ν ) /ϵ 2 ) iterations for a max-edge weak learner.
Revised: O (ln( m/ν ) /ϵ 2 ) iterations for a max-edge weak learner.

Original: Similarly, one can easily verify that ERLPBoost and C-ERLPBoost are also instances of the Frank-Wolfe algorithm.
Revised: Similarly, one can easily verify that ERLPBoost and C-ERLPBoost are also instances of the Frank-Wolfe algorithm.

Original: See the proof in appendix.
Revised: See the proof in appendix.

Original: Theorem 1 LPBoost, ERLPBoost, and C-ERLPBoost are instances of the Frank-Wolfe algorithm.
Revised: Theorem 1 LPBoost, ERLPBoost, and C-ERLPBoost are instances of the Frank-Wolfe algorithm.

Original: 3.2. A generic scheme for margin-maximizing boosting
Revised: 3.2. A generic scheme for margin-maximizing boosting

Original: Input: A matrix A
Revised: Input: A matrix A

Original: =
Revised: =

Original: ( y i h
Revised: ( y i h

Original: j ( x i ))
Revised: j ( x i ))

Original: i,j ∈ [ − 1 , +1] m × n and a set of basis vectors E t +1 ⊂ P n .
Revised: i,j ∈ [ − 1 , +1] m × n and a set of basis vectors E t +1 ⊂ P n .

Original: Output: w ← arg max w ∈ CH( E t +1 ) min d ∈P m ν d ⊤ A w .
Revised: Output: w ← arg max w ∈ CH( E t +1 ) min d ∈P mν d ⊤ A w .

Original: Algorithmt
Revised: Algorithm

Original: 1: w (1) t +1 =
Revised: 1: w (1) t +1 =

Original: w t + λ t ( e j t +1 − w t ), where λ t = min (cid:26) 1 , max (cid:26) 0 , d ⊤ t A ( e j +1 − w t ) η
Revised: w t + λ t ( e j t +1 − w t ), where λ t = min (cid:26) 1 , max (cid:26) 0 , d ⊤ t A ( e j +1 − w t ) η

Original: ∥ A ( e j +1 − w t ) ∥ 2 ∞ (cid:27)(cid:27) .
Revised: ∥ A ( e j +1 − w t ) ∥ 2 ∞ (cid:27)(cid:27) .

Original: We propose a Frank-Wolfe-like boosting scheme, shown in algorithm 1.
Revised: We propose a Frank-Wolfe-like boosting scheme, shown in algorithm 1.

Original: Intuitively, the
Revised: Intuitively, the

Original: Frank-Wolfe rule F is a safety net for the convergence guarantee.
Revised: Frank-Wolfe rule F is a safety net for the convergence guarantee.

Original: Further, the convergence analysis only depends on the Frank-Wolfe rule w (1) t +1 so one can incorporate any update rule to B .
Revised: Further, the convergence analysis only depends on the Frank-Wolfe rule w (1) t +1 so one can incorporate any update rule to B .

Original: For example, one can use w E t +1 as B ( A, E t +1 ).
Revised: For example, one can use w E t +1 as B ( A, E t +1 ).

Original: Algorithm 1 becomes ERLPBoost in this case since w t +1 = w (2) t +1 holds for all t .
Revised: Algorithm 1 becomes ERLPBoost in this case since w t +1 = w (2) t +1 holds for all t .

Original: Recall that our primary objective is to find a weight vector w that optimizes the soft margin, formulated as LP.
Revised: Recall that our primary objective is to find a weight vector w that optimizes the soft margin, formulated as LP.

Original: The most practical algorithm, LPBoost, solves the optimization problem over past hypotheses, so using the solution as B is a natural choice.
Revised: The most practical algorithm, LPBoost, solves the optimization problem over past hypotheses, so using the solution as B is a natural choice.

Original: Algorithmsummarizes this update.
Revised: Algorithmsummarizes this update.

Original: As described in [7], one can compute the distribution d t = ∇ ˜
Revised: As described in [7], one can compute the distribution d t = ∇ ˜

Original: f ⋆ ( − A w t ) by a sorting-based algorithm, which takes O ( m ln m ) iterations 1 .
Revised: f ⋆ ( − A w t ) by a sorting-based algorithm, which takes O ( m ln m ) iterations 1 .

Original: Thus, the time complexity of our scheme depends on the secondary algorithm B .
Revised: Thus, the time complexity of our scheme depends on the secondary algorithm B .

Original: The following theorem guarantees the convergence rate for algorithm 1.
Revised: The following theorem guarantees the convergence rate for algorithm 1.

Original: We give the proof in the appendix.
Revised: We give the proof in the appendix.

Original: Theorem 2 (A convergence rate for algorithm 1) Assume that the weak learner returns a hypothesis h
Revised: Theorem 2 (A convergence rate for algorithm 1) Assume that the weak learner returns a hypothesis h

Original: j t +1 ∈ H that satisfies ( d ⊤ t A )
Revised: j t +1 ∈ H that satisfies ( d ⊤ t A )

Original: j t +1 ≥ g for some unknown guarantee g .
Revised: j t +1 ≥ g for some unknown guarantee g .

Original: Let F be a Frank-Wolfe rule with classic step λ t = 2 t +2 , or short-step as in algorithm 3.
Revised: Let F be a Frank-Wolfe rule with classic step λ t = 2 t +2 , or short-step as in algorithm 3.

Original: Then, for any secondary algorithm B , algorithm 1 outputs H T = P
Revised: Then, for any secondary algorithm B , algorithm 1 outputs H T = P

Original: Tt =1 w j t
Revised: Tt =1 w j t

Original: h
Revised: h

Original: j t satisfying (1) in O (cid:16) 1 ϵ 2 ln mν (cid:17) iterations.
Revised: j t satisfying (1) in O (cid:16) 1 ϵ 2 ln mν (cid:17) iterations.

Original: 4. Experiments
Revised: 4. Experiments

Original: We compared the boosting algorithms on Gunnar Rätsch’s benchmark dataset 2 .
Revised: We compared the boosting algorithms on Gunnar Rätsch’s benchmark dataset 2 .

Original: We call our scheme with secondary algorithm 2 as MLPBoost.
Revised: We call our scheme with secondary algorithm 2 as MLPBoost.

Original: MLPB.
Revised: MLPB.

Original: (SS) uses the short-step FrankWolfe algorithm 3.
Revised: (SS) uses the short-step FrankWolfe algorithm 3.

Original: As table 2 shows, MLPB.
Revised: As table 2 shows, MLPB.

Original: (SS) terminates much faster than C-ERLPB.
Revised: (SS) terminates much faster than C-ERLPB.

Original: This result indicates that the secondary update significantly improves the objective.
Revised: This result indicates that the secondary update significantly improves the objective.

Original: [Table]
Revised: [Table]

Original: The worst case for LPB.
Revised: The worst case for LPB.

Original: Although LPB.
Revised: Although LPB.

Original: outperforms the running time in Table 2, it takes m/ 2 iterations for the worst case [8].
Revised: outperforms the running time in Table [Table] it takes m/ 2 iterations for the worst case [8].

Original: Even in this case, the other algorithms terminate in 2 iterations.
Revised: Even in this case, the other algorithms terminate in 2 iterations.

Original: Appendix A. Fenchel duality review
Revised: Appendix A. Fenchel duality review

Original: First we introduce some additional notations.
Revised: First we introduce some additional notations.

Original: For a function f : R m → ( −∞ , + ∞ ], let dom f
Revised: For a function f : R m → ( −∞ , + ∞ ], let dom f

Original: = { d | f ( d ) < + ∞} .
Revised: = { d | f ( d ) < + ∞} .

Original: For a convex set C ⊂ R n , define the set of interior points int( C ) as
Revised: For a convex set C ⊂ R n , define the set of interior points int( C ) as

Original: [Equation]
Revised: [Equation]

Original: For sets A and B over R n , define A − B = { a − b | a ∈ A, b ∈ B } .
Revised: For sets A and B over R n , define A − B = { a − b | a ∈ A, b ∈ B } .

Original: Theorem 3 ([1]) Let f : R m → ( −∞ , + ∞ ] and g : R n → ( −∞ , + ∞ ] be convex functions and a linear map M : R m → R n .
Revised: Theorem 3 ([1]) Let f : R m → ( −∞ , + ∞ ] and g : R n → ( −∞ , + ∞ ] be convex functions and a linear map M : R m → R n .

Original: Define the Fenchel problems
Revised: Define the Fenchel problems

Original: [Equation]
Revised: [Equation]

Original: Then, γ ≥ ρ holds.
Revised: Then, γ ≥ ρ holds.

Original: Further, γ = ρ holds if
Revised: Further, γ = ρ holds if

Original: [Equation]
Revised: [Equation]

Original: Further, points d ⋆ ∈ P mν and w ⋆ ∈ P n are optimal solutions for problems (2) and (3), respectively, if and only if − M w ⋆ ∈
Revised: Further, points d ⋆ ∈ P mν and w ⋆ ∈ P n are optimal solutions for problems (2) and (3), respectively, if and only if − M w ⋆ ∈

Original: ∂f ( d ⋆ ) and w ⋆ ∈ ∂g ( M ⊤ d ⋆ ) .
Revised: ∂f ( d ⋆ ) and w ⋆ ∈ ∂g ( M ⊤ d ⋆ ) .

Original: The following lemma is useful to prove our result.
Revised: The following lemma is useful to prove our result.

Original: Lemma 4 Let f, ˜ f : R m → ( −∞ , + ∞ ] be functions such that ∃ c > 0 , ∀ θ , f ( θ ) ≤ ˜ f
Revised: Lemma 4 Let f, ˜ f : R m → ( −∞ , + ∞ ] be functions such that ∃ c > 0 , ∀ θ , f ( θ ) ≤ ˜ f

Original: ( θ ) ≤ f ( θ )
Revised: ( θ ) ≤ f ( θ )

Original: + c .
Revised: + c .

Original: Then, f ⋆ ( µ ) − c ≤ ˜ f
Revised: Then, f ⋆ ( µ ) − c ≤ ˜ f

Original: ⋆ ( µ ) ≤ f ⋆ ( µ ) holds for all µ .
Revised: ⋆ ( µ ) ≤ f ⋆ ( µ ) holds for all µ .

Original: Appendix B. Technical Lemmas
Revised: Appendix B. Technical Lemmas

Original: The following lemma shows the maximum value of the relative entropy from the uniform distribution over the capped probability simplex P mν .
Revised: The following lemma shows the maximum value of the relative entropy from the uniform distribution over the capped probability simplex P mν .

Original: Lemma 5 max d ∈P mν ∆( d ) ≤ ln mν .
Revised: Lemma 5 max d ∈P mν ∆( d ) ≤ ln mν .

Original: Proof Since the relative entropy from the uniform distribution achieves its maximal value at the extreme points of P mν , a maximizer has the form
Revised: Proof Since the relative entropy from the uniform distribution achieves its maximal value at the extreme points of P mν , a maximizer has the form

Original: [Equation]
Revised: [Equation]

Original: for some k ∈ [ m ].
Revised: for some k ∈ [ m ].

Original: Plugging the maximizer into ∆( d ), we can write the objective function as ∆( d ) =
Revised: Plugging the maximizer into ∆( d ), we can write the objective function as ∆( d ) =

Original: ( k/ν ) ln( m/ν ) + s ln( sm ).
Revised: ( k/ν ) ln( m/ν ) + s ln( sm ).

Original: If s = 0, ∆( d ) =
Revised: If s = 0, ∆( d ) =

Original: ln( m/ν ) holds since k = ν .
Revised: ln( m/ν ) holds since k = ν .

Original: If s > 0, k/ν < 1 so that
Revised: If s > 0, k/ν < 1 so that

Original: [Equation]
Revised: [Equation]

Original: The next lemma shows the dual problem of the edge minimization.
Revised: The next lemma shows the dual problem of the edge minimization.

Original: Lemma 6 Let f : R m → { 0 , + ∞} , g : R n → R be functions defined as
Revised: Lemma 6 Let f : R m → { 0 , + ∞} , g : R n → R be functions defined as

Original: [Equation]
Revised: [Equation]

Original: Then, the dual problem of edge minimization
Revised: Then, the dual problem of edge minimization

Original: [Equation]
Revised: [Equation]

Original: is the soft margin maximization
Revised: is the soft margin maximization

Original: [Equation]
Revised: [Equation]

Original: Further, the strong duality holds.
Revised: Further, the strong duality holds.

Original: Proof We can use Theorem 3 to derive the dual problem.
Revised: Proof We can use Theorem 3 to derive the dual problem.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: you can verify the dual form becomes as in (6).
Revised: you can verify the dual form becomes as in (6).

Original: To prove the strong duality, it is enough to prove eq.
Revised: To prove the strong duality, it is enough to prove eq.

Original: By definition, dom g = R n and dom f = P mν and hence
Revised: By definition, dom g = R n and dom f = P mν and hence

Original: [Equation]
Revised: [Equation]

Original: Obviously, 0 ∈ int(dom g − A ⊤ dom f ) and thus the strong duality holds.
Revised: Obviously, 0 ∈ int(dom g − A ⊤ dom f ) and thus the strong duality holds.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we can write the dual problem (6) explicitly:
Revised: we can write the dual problem (6) explicitly:

Original: [Equation]
Revised: [Equation]

Original: By the same derivation, we get the dual problem for the regularized edge minimization problem.
Revised: By the same derivation, we get the dual problem for the regularized edge minimization problem.

Original: Corollary 7 Let f, g be the functions defined in Lemma 6 and let ∆( d ) =
Revised: Corollary 7 Let f, g be the functions defined in Lemma 6 and let ∆( d ) =

Original: P mi =1 d i ln d
Revised: P mi =1 d i ln d

Original: i + ln( m ) be the relative entropy function from the uniform distribution.
Revised: i + ln( m ) be the relative entropy function from the uniform distribution.

Original: Define ˜ f
Revised: Define ˜ f

Original: = f + (1 /η )
Revised: = f + (1 /η )

Original: ∆ for some η > 0 .
Revised: ∆ for some η > 0 .

Original: Then, the dual problem of
Revised: Then, the dual problem of

Original: [Equation]
Revised: [Equation]

Original: Appendix C. Proofs
Revised: Appendix C. Proofs

Original: This section is dedicated for the proof of the main result.
Revised: This section is dedicated for the proof of the main result.

Original: We first show the fact that
Revised: We first show the fact that

Original: LPBoost and ERLPBoost are also instances of FW.
Revised: LPBoost and ERLPBoost are also instances of FW.

Original: Before getting into the proof, we first review the Frank-Wolfe algorithm.
Revised: Before getting into the proof, we first review the Frank-Wolfe algorithm.

Original: C.1. The Frank-Wolfe algorithms
Revised: C.1. The Frank-Wolfe algorithms

Original: The original Frank-Wolfe (FW) algorithm is a first-order iterative algorithm invented by [3].
Revised: The original Frank-Wolfe (FW) algorithm is a first-order iterative algorithm invented by [3].

Original: The FW algorithm solves the problems of the form: min x ∈C f ( x ), where C ⊂ R m is a closed convex set and
Revised: The FW algorithm solves the problems of the form: min x ∈C f ( x ), where C ⊂ R m is a closed convex set and

Original: f : C → R is an η -smooth and convex function.
Revised: f : C → R is an η -smooth and convex function.

Original: In each step t , FW seeks an extreme point s t +1 ∈ arg min s ∈C s ⊤ ∇ f ( x t ).
Revised: In each step t , FW seeks an extreme point s t +1 ∈ arg min s ∈C s ⊤ ∇ f ( x t ).

Original: Then, it updates the iterate as x t +1 = x t + λ t ( s t +1 − x t ) for some λ t ∈ [0 , 1].
Revised: Then, it updates the iterate as x t +1 = x t + λ t ( s t +1 − x t ) for some λ t ∈ [0 , 1].

Original: Although the classical result [3, 5] suggests λ t = 2 / ( t + 2), there are many choices of λ t .
Revised: Although the classical result [3, 5] suggests λ t = 2 / ( t + 2), there are many choices of λ t .

Original: For example, one can choose λ t as λ t :=
Revised: For example, one can choose λ t as λ t :=

Original: clip [0 , 1] ( x t − s t +1 )
Revised: clip [0 , 1] ( x t − s t +1 )

Original: ⊤ ∇ f ( x t ) η
Revised: ⊤ ∇ f ( x t ) η

Original: ∥ s t +1 − x t ∥, where clip [0 , 1] x = max { 0 , min { 1 , x }} .
Revised: ∥ s t +1 − x t ∥, where clip [0 , 1] x = max { 0 , min { 1 , x }} .

Original: This is the optimal solution that minimizes the RHS of the strongly-smooth bound and is often called a short-step strategy.
Revised: This is the optimal solution that minimizes the RHS of the strongly-smooth bound and is often called a short-step strategy.

Original: The FW algorithms converge to an ϵ -approximate solution in
Revised: The FW algorithms converge to an ϵ -approximate solution in

Original: O ( η/ϵ ) iterations if the objective function is η -smooth w.r.t.
Revised: O ( η/ϵ ) iterations if the objective function is η -smooth w.r.t.

Original: some norm over C [3, 5].
Revised: some norm over C [3, 5].

Original: The best advantage of the FW algorithm is the projection-free property; there is no projection onto C , so the running time per step is faster than the projected gradient methods.
Revised: The best advantage of the FW algorithm is the projection-free property; there is no projection onto C , so the running time per step is faster than the projected gradient methods.

Original: C.2. Proof of Theorem
Revised: C.2. Proof of Theorem

Original: LPBoost and ERLPBoost update the distribution as
Revised: LPBoost and ERLPBoost update the distribution as

Original: [Equation]
Revised: [Equation]

Original: By definition of the Fenchel problems, these distribution correspond to the (sub)gradient vectors d
Revised: By definition of the Fenchel problems, these distribution correspond to the (sub)gradient vectors d

Original: L t ∈ ∂f ⋆ ( θ L t ) and d E t = ∇ ˜
Revised: L t ∈ ∂f ⋆ ( θ L t ) and d E t = ∇ ˜

Original: f ⋆ ( θ E t ), where θ L t
Revised: f ⋆ ( θ E t ), where θ L t

Original: [Equation]
Revised: [Equation]

Original: − A w Lt and θ E t = − A w Et are
Revised: − A w Lt and θ E t = − A w Et are

Original: [Equation]
Revised: [Equation]

Original: These properties are the immediate consequence of Fenchel duality theorem.
Revised: These properties are the immediate consequence of Fenchel duality theorem.

Original: Thus, we can say that LPBoost and ERLPBoost are instances of the fully-corrective Frank-Wolfe algorithm.
Revised: Thus, we can say that LPBoost and ERLPBoost are instances of the fully-corrective Frank-Wolfe algorithm.

Original: C.3. Proof of our scheme
Revised: C.3. Proof of our scheme

Original: We prove the convergence guarantee for algorithm 1.
Revised: We prove the convergence guarantee for algorithm 1.

Original: Before getting into the main result, we first justify the stopping criterion.
Revised: Before getting into the main result, we first justify the stopping criterion.

Original: Lemma
Revised: Lemma

Original: Let ϵ t :=
Revised: Let ϵ t :=

Original: min 0 ≤ τ ≤ t ( d ⊤ τ A )
Revised: min 0 ≤ τ ≤ t ( d ⊤ τ A )

Original: j τ +1 + ˜
Revised: j τ +1 + ˜

Original: f
Revised: f

Original: ⋆ ( − A w t ) be the optimality gap defined in algorithm 2 and let η = 2 ln( m/ν ) /ϵ .
Revised: ⋆ ( − A w t ) be the optimality gap defined in algorithm 2 and let η = 2 ln( m/ν ) /ϵ .

Original: Then, ϵ t ≤ ϵ/ 2 implies − f ⋆ ( − A w t ) ≥ g − ϵ .
Revised: Then, ϵ t ≤ ϵ/ 2 implies − f ⋆ ( − A w t ) ≥ g − ϵ .

Original: Proof By the weak-learnability assumption, ϵ t ≥ g + ˜
Revised: Proof By the weak-learnability assumption, ϵ t ≥ g + ˜

Original: f
Revised: f

Original: ⋆ ( − A w t ).
Revised: ⋆ ( − A w t ).

Original: The statement follows from Lemma 4.
Revised: The statement follows from Lemma 4.

Original: Note that this criterion is better than the one of C-ERLPBoost.
Revised: Note that this criterion is better than the one of C-ERLPBoost.

Original: The following theorem shows the convergence rate for our scheme.
Revised: The following theorem shows the convergence rate for our scheme.

Original: Theorem 9 (Recap.)
Revised: Theorem 9 (Recap.)

Original: Assume that the weak learner returns a hypothesis h
Revised: Assume that the weak learner returns a hypothesis h

Original: j t +1 ∈ H that satisfies ( d ⊤ t A )
Revised: j t +1 ∈ H that satisfies ( d ⊤ t A )

Original: j t +1 ≥ g for some unknown guarantee g .
Revised: j t +1 ≥ g for some unknown guarantee g .

Original: Let F be a FW update with classic step λ t = 2 t +2 , or short-step as in algorithm 3.
Revised: Let F be a FW update with classic step λ t = 2 t +2 , or short-step as in algorithm 3.

Original: Then, for any secondary algorithm B , algorithm 1 outputs H T = P
Revised: Then, for any secondary algorithm B , algorithm 1 outputs H T = P

Original: Tt =1 w j t
Revised: Tt =1 w j t

Original: h
Revised: h

Original: j t satisfying (1) in O (cid:16) 1 ϵ 2 ln mν (cid:17) iterations.
Revised: j t satisfying (1) in O (cid:16) 1 ϵ 2 ln mν (cid:17) iterations.

Original: Proof First of all, we prove the bound for the classic step size.
Revised: Proof First of all, we prove the bound for the classic step size.

Original: We start by showing the recursion
Revised: We start by showing the recursion

Original: [Equation]
Revised: [Equation]

Original: By using the definition of w t +1 and the η -smoothness of ˜ f
Revised: By using the definition of w t +1 and the η -smoothness of ˜ f

Original: ⋆ ,
Revised: ⋆ ,

Original: [Equation]
Revised: [Equation]

Original: where eq.
Revised: where eq.

Original: (8) holds since A ∈ [ − 1 , +1] m × n and e j t +1 , w t ∈ P n .
Revised: (8) holds since A ∈ [ − 1 , +1] m × n and e j t +1 , w t ∈ P n .

Original: By the non-negativity of the entropy function and the definition of d t , we get
Revised: By the non-negativity of the entropy function and the definition of d t , we get

Original: [Equation]
Revised: [Equation]

Original: Combining eq.
Revised: Combining eq.

Original: (8) and (9), we obtain (7).
Revised: (8) and (9), we obtain (7).

Original: Now, we prove the following inequality by induction on t .
Revised: Now, we prove the following inequality by induction on t .

Original: [Equation]
Revised: [Equation]

Original: For the base case t = 1, the inequality (10) holds; ϵ 1 ≤ (1 − λ 0 )
Revised: For the base case t = 1, the inequality (10) holds; ϵ 1 ≤ (1 − λ 0 )

Original: ϵ 

Revised: ϵ

Original: + 2 ηλ 20 = 2 η ≤ 8 η 0+2 .
Revised: + 2 ηλ 20 = 2 η ≤ 8 η 0+2 .

Original: Assume that (10) holds for t ≥ 1.
Revised: Assume that (10) holds for t ≥ 1.

Original: By the inductive assumption,
Revised: By the inductive assumption,

Original: [Equation]
Revised: [Equation]

Original: Therefore, (10) holds for all t ≥ 1.
Revised: Therefore, (10) holds for all t ≥ 1.

Original: By definition of η , ϵ T ≤ ϵ 2 holds after T ≥ 32 ϵ 2 ln mν − 2 iterations.
Revised: By definition of η , ϵ T ≤ ϵ 2 holds after T ≥ 32 ϵ 2 ln mν − 2 iterations.

Original: Lemma 8 yields the convergence rate.
Revised: Lemma 8 yields the convergence rate.

Original: For the short-step case, we get a similar recursion:
Revised: For the short-step case, we get a similar recursion:

Original: [Equation]
Revised: [Equation]

Original: Optimizing λ in RHS and applying the inequality (9), we get ϵ t − ϵ t +1 ≥ ϵ 2 t / 8 η .
Revised: Optimizing λ in RHS and applying the inequality (9), we get ϵ t − ϵ t +1 ≥ ϵ 2 t / 8 η .

Original: With this inequality, one can easily verify that the same iteration bound (10) holds for this case.
Revised: With this inequality, one can easily verify that the same iteration bound (10) holds for this case.

Original: 1: Let w t =
Revised: 1: Let w t =

Original: P e ∈ E t α t, e e be the current representation of w t w.r.t.
Revised: P e ∈ E t α t, e e be the current representation of w t w.r.t.

Original: the basis vectors E t ⊂ E t with positive coefficients { α t, e } e ∈ E t .
Revised: the basis vectors E t ⊂ E t with positive coefficients { α t, e } e ∈ E t .

Original: 2: Compute an away basis e Away ∈ arg min e ∈ E t d ⊤ t
Revised: 2: Compute an away basis e Away ∈ arg min e ∈ E t d ⊤ t

Original: A e and set λ t, max = α t, e Away .
Revised: A e and set λ t, max = α t, e Away .

Original: 3: Compute the step size λ t ← arg min λ ∈ [0 ,λ t, max ] ˜ f
Revised: 3: Compute the step size λ t ← arg min λ ∈ [0 ,λ t, max ] ˜ f

Original: ⋆ ( − A ( w t + λ ( e t +1 − e Away ))).
Revised: ⋆ ( − A ( w t + λ ( e t +1 − e Away ))).

Original: Output: w (1) t +1 =
Revised: Output: w (1) t +1 =

Original: w t + λ t ( e j t +1 − e Away ).
Revised: w t + λ t ( e j t +1 − e Away ).

Original: C.4. Pairwise rule
Revised: C.4. Pairwise rule

Original: The FW update rule w (1) t +1 of algorithm 3 comes from the FW algorithm with short-step sizes.
Revised: The FW update rule w (1) t +1 of algorithm 3 comes from the FW algorithm with short-step sizes.

Original: One can apply other update rules as F .
Revised: One can apply other update rules as F .

Original: The pairwise Frank-Wolfe (PFW) is the one of a state-of-the-art Frank-Wolfe algorithm [6].
Revised: The pairwise Frank-Wolfe (PFW) is the one of a state-of-the-art Frank-Wolfe algorithm [6].

Original: The basic idea of PFW is to move the weight from the most worthless hypothesis to the newly attained one.
Revised: The basic idea of PFW is to move the weight from the most worthless hypothesis to the newly attained one.

Original: We can derive a similar bound for this case.
Revised: We can derive a similar bound for this case.

Original: See appendix.
Revised: See appendix.

Original: As described in the main section, one can apply the other variants of the FW algorithm.
Revised: As described in the main section, one can apply the other variants of the FW algorithm.

Original: Algorithm 4 summarizes this variant.
Revised: Algorithm 4 summarizes this variant.

Original: Appendix D. Additional experiments
Revised: Appendix D. Additional experiments

Original: This section includes experiments, not in the main paper.
Revised: This section includes experiments, not in the main paper.

Original: MLPB.
Revised: MLPB.

Original: (PFW) in the figures and tables corresponds to MLPBoost with the PFW algorithm 4.
Revised: (PFW) in the figures and tables corresponds to MLPBoost with the PFW algorithm 4.

Original: Settings.
Revised: Settings.

Original: We set the capping parameters ν ∈ { pm | p = 0 .
Revised: We set the capping parameters ν ∈ { pm | p = 0 .

Original: . , 0 . 5 } and the tolerance parameter ϵ = 0 .
Revised: . , 0 . 5 } and the tolerance parameter ϵ = 0 .

Original: 01, where m is the number of training instances.
Revised: 01, where m is the number of training instances.

Original: We use the weak learner that returns the best decision tree of depth 2.
Revised: We use the weak learner that returns the best decision tree of depth 2.

Original: We measure the CPU time and the System time using /usr/bin/time -v command.
Revised: We measure the CPU time and the System time using /usr/bin/time -v command.

Original: We measure the running time with capping parameters over the capping parameters for each dataset and took their average.
Revised: We measure the running time with capping parameters over the capping parameters for each dataset and took their average.

Original: Tableshows the results.
Revised: Tableshows the results.

Original: As this table indicates, MLPB.
Revised: As this table indicates, MLPB.

Original: (SS) is faster than ERLPB.
Revised: (SS) is faster than ERLPB.

Original: We first show the test error comparison.
Revised: We first show the test error comparison.

Original: We performed 5-fold cross validation to find the best capping parameter ν ∈ [1 , m ].
Revised: We performed 5-fold cross validation to find the best capping parameter ν ∈ [1 , m ].

Original: After finding the best one, we trained the model on overall dataset and measure the test data.
Revised: After finding the best one, we trained the model on overall dataset and measure the test data.

Original: Table 3 shows the result.
Revised: Table 3 shows the result.

Original: Our algorithm somehow achieves the smallest test error for most dataset.
Revised: Our algorithm somehow achieves the smallest test error for most dataset.

Original: Next we show the time comparison for all dataset.
Revised: Next we show the time comparison for all dataset.

Original: Table 4 summarizes the result.
Revised: Table 4 summarizes the result.

Original: Figure 1 shows the convergence curve.
Revised: Figure 1 shows the convergence curve.

Original: As expected, our algorithm converges faster than
Revised: As expected, our algorithm converges faster than

Original: ERLPBoost and is competitive with LPBoost.
Revised: ERLPBoost and is competitive with LPBoost.

Original: Further, we compare the test error decrease.
Revised: Further, we compare the test error decrease.

Original: Figure 2 shows the test error curves.
Revised: Figure 2 shows the test error curves.

Original: Our algorithm somehow achieves low test errors in most datasets.
Revised: Our algorithm somehow achieves low test errors in most datasets.

Original: Now, we compare the Frank-Wolfe algorithms, SS and PFW.
Revised: Now, we compare the Frank-Wolfe algorithms, SS and PFW.

Original: SS indicates the short-steps strategy, and PFW indicates the pairwise strategy.
Revised: SS indicates the short-steps strategy, and PFW indicates the pairwise strategy.

Original: Figure 3 shows the number of iterations that adopt w (2) t +1 as the next iterate.
Revised: Figure 3 shows the number of iterations that adopt w (2) t +1 as the next iterate.

Original: Finally, we verify the effectiveness of the secondary rule, w (2) t +1 .
Revised: Finally, we verify the effectiveness of the secondary rule, w (2) t +1 .

Original: For comparison, we measured the soft margin objective and time for MLPB.
Revised: For comparison, we measured the soft margin objective and time for MLPB.

Original: (SS), MLPB.
Revised: (SS), MLPB.

Original: (PFW), FW, and
Revised: (PFW), FW, and

Original: PFW.
Revised: PFW.

Original: FW is the classic FW algorithm with short-step strategy and PFW is the Pairwise
Revised: FW is the classic FW algorithm with short-step strategy and PFW is the Pairwise

Original: FW algorithm.
Revised: FW algorithm.

Original: Note that FW is equivalent to C-ERLPBoost.
Revised: Note that FW is equivalent to C-ERLPBoost.

Original: Figure 4 shows the curve.
Revised: Figure 4 shows the curve.

Original: As this figure shows, the secondary rule, w (2) t +1 improves the objective value significantly.
Revised: As this figure shows, the secondary rule, w (2) t +1 improves the objective value significantly.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: m a n j c t i v
Revised: m a n j c t i v

Original: [Figure]
Revised: [Figure]

Original: S o f t m a r g i n o b
Revised: S o f t m a r g i n o b

Original: j e c t i v e
Revised: j e c t i v e

Original: Twonorm
Revised: Twonorm

Original: LPB.ERLPB.MLPB.
Revised: LPB.ERLPB.MLPB.

Original: (SS) MLPB.
Revised: (SS) MLPB.

Original: (PFW) FW
Revised: (PFW) FW

Original: [Figure]
Revised: [Figure]

Original: tends to decrease the test error.
Revised: tends to decrease the test error.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]