Active Learning of Classifiers with Label and Seed Queries
Abstract
We study exact active learning of binary and multiclass classifiers with margin.
Given an n -point set X ⊂ R m , we want to learn an unknown classifier on X whose classes have finite strong convex hull margin , a new notion extending the
SVM margin.
In the standard active learning setting, where only label queries are allowed, learning a classifier with strong convex hull margin γ requires in the worst case Ω (cid:0) 1 + 1 γ (cid:1) m − 1 2 queries.
On the other hand, using the more powerful seed queries (a variant of equivalence queries), the target classifier could be learned in O ( m log n ) queries via Littlestone’s Halving algorithm; however, Halving is computationally inefficient.
In this work we show that, by carefully combining the two types of queries, a binary classifier can be learned in time poly( n + m ) using only O ( m 2 log n ) label queries and O (cid:0) m log mγ (cid:1) seed queries; the result extends to k -class classifiers at the price of a k !
k 2 multiplicative overhead.
Similar results hold when the input points have bounded bit complexity, or when only one class has strong convex hull margin against the rest.
We complement the upper bounds by showing that in the worst case any algorithm needs Ω (cid:0) km log 1 γ (cid:1) seed and label queries to learn a k -class classifier with strong convex hull margin γ .
Introduction
This work investigates efficient algorithms for exact active learning of binary and multiclass classifiers in the transductive setting.
Given a set X of n points in R m , our goal is to learn a function h : X → [ k ] belonging to some class H .
In the classic active learning framework, h identifies a subset of X , and the algorithm learns h via queries LABEL ( x ) that return h ( x ) for any given x ∈ X .
In that case, it is well-known that h can be learned with O (log n ) LABEL queries if the star number of H is finite [Hanneke and Yang, 2015].
Unfortunately, even simple families such as linear classifiers have unbounded star number, in which case Ω( n ) LABEL queries are needed in the worst case.
To bypass this lower bound, it has become increasingly common to introduce enriched queries , that reveal additional information on h and are plausible in practice.
One notable example is that of comparison queries for linear separators in R m which, given any pair of points x, y ∈ X , reveal which one is closer to the decision boundary.
As proven by Kane et al.
[2017], under some margin assumptions the combination of LABEL and comparisons yields exponential savings, allowing one to learn linear separators with only O (log n ) queries.
In this work we combine LABEL queries with seed queries .
For any U ⊂ X and any i ∈ [ k ] , a query SEED ( U, i ) returns an abitrary point x in U ∩ C i , where
C i = h − 1 ( i ) , or NIL if no such x exists.
SEED queries are natural in certain settings like crowdsourcing—e.g., finding the image of a car, see also Beygelzimer et al.
[2016]—and have been used implicitly or explicitly in several works [Hanneke, 2009, Balcan and Hanneke, 2012, Attenberg and Provost, 2010, Tong and Chang, 2001, Doyle et al., 2011, Bressan et al., 2021b].
It is not hard to see that, using SEED alone, one can implement Littlestone’s Halving algorithm and learn any h ∈ H with O (log |H| ) queries 1 .
For instance, linear separators in R m can be learned with O ( m log n ) SEED queries.
The catch is that, save for special cases, it is not known how to run the Halving algorithm in polynomial time.
Therefore, using SEED to obtain a computationally efficient active learning algorithm is less trivial than it seems at first glance.
The goal of this work is understanding whether one can actively learn binary and multiclass classifiers efficiently by using LABEL and SEED queries together.
In line with Kane et al.
[2017] and other previous works, we make assumptions on H .
Our main assumption is that every class C i has strong convex hull margin γ > 0 .
This means that, for any j ̸ = i , C i and C j are linearly separable with a margin that is at least γ 2 times the diameter of C i .
Moreover, it is sufficient that this hold under some pseudometric d i , unknown to the learner, that is homogeneous and invariant under translation (i.e., induced by a seminorm).
This gives to every class its own personalized notion of distance that can be sensitive to the “scale” of the class.
This assumption strictly generalizes the classical SVM margin; and, when suitably generalized, it captures stability properties of center-based clusterings Awasthi et al.
[2012], Bilu and Linial [2012].
Using LABEL alone, Bressan et al.
[2021a] showed that learning a multiclass classifier with (strong) convex hull margin γ > 0 requires between Ω (cid:0) 1 + 1 γ (cid:1) ( m − 1) / 2 and ˜ O (cid:0) k 3 m 5 (cid:0) 1 + 1 γ (cid:1)
m log n (cid:1) queries.
This exponential dependence on m implies that, unless m ≪ log n/ log 1 γ , one needs Θ( n ) LABEL queries in the worst case.
On the other hand our margin implies linear separability and thus, as noted above, a O ( m log n ) SEED query bound for the binary case, but with a running time that can be superpolynomial.
This leaves open the following problem, which is the subject of this work:
Can one learn a multiclass classifier h with strong convex hull margin
γ > 0 on X ⊂ R m in time poly( n + m ) using a number of queries that grows polynomially with m ?
We solve the above question in the affirmative by proving that, with a careful combination of LABEL and SEED queries, one can do much better than using either query in isolation.
For binary classification ( k = 2 ), we show:
Theorem 1.
Any binary classifier h with strong convex hull margin γ > 0 over X ⊂ R m can be learned in time poly( n + m ) using in expectation O ( m 2 log n ) LABEL queries and O (cid:0) m log mγ (cid:1)
SEED queries.
Note that, unless γ is exceedingly small, Theorem 1 uses far fewer SEED than LABEL queries, which is a strength since SEED is arguably more expensive to implement.
For instance, if γ = Ω(1 / poly( m )) then we use O ( m 2 log n ) LABEL queries but only O ( m log m ) SEED queries.
To prove Theorem 1 we design a novel algorithm that works in two phases.
The first phase learns what we call an α -rounding of X w.r.t.
h .
Loosely speaking, this is a partition ( X 1 , X 2 ) of X such that each X i lies inside α conv( C i ) where conv( C i ) is the convex hull of C i (see below for the formal definition).
We show that, in polynomial time and using O ( m 2 log n ) LABEL queries, one can compute an α -rounding of X i for α
= O ( m 3 ) .
This allows us to put X i in near-isotropic position so that X i has radius 1 and to separate C 1 ∩
X i from C 2 ∩
X i with margin η = Ω( γ/m 3 ) .
In the second phase, the algorithm uses SEED to implement a cutting plane algorithm that learns C 1 ∩ X i and C
∩
X i using
O (cid:0) m log 1 η (cid:1) =
O (cid:0) m log mγ (cid:1) queries in time poly( n + m ) .
Using a recursive approach, Theorem 1 can be extended to k > 2 at the price of a k !
k 2 multiplicative overhead:
Theorem 2.
Any k -class classifier h with strong convex hull margin γ > 0 over X ⊂ R m can be learned in time poly( n + m ) using in expectation O
( k !
k 2 m 2 log n ) LABEL queries and
O (cid:0) k !
k 2 m log mγ (cid:1) SEED queries.
We also consider the case where only one class has strong convex hull margin against the rest of the points w.r.t.
a metric d induced by a norm ∥ · ∥ d .
In this case we obtain a bound parameterized by the distortion κ d of d (see Section 1.1):
Theorem 3.
Suppose C ⊂ X has strong convex hull margin γ ∈ (0 , 1] w.r.t.
a metric d with distortion κ d < ∞ .
Given only X , one can learn C in time poly( n + m ) using O (log n ) LABEL queries and
O (cid:0) m log κ d γ (cid:1) SEED queries in expectation.
As an application of our cutting-plane algorithm we also show that one can learn a k -class classifier whose classes are pairwise linearly separable in time poly( n + m ) using, in expectation, O ( k 2 m 3 B )
SEED queries if every x ∈ X has rational coordinates that can be encoded in B bits, and O ( k 2 m ( B + m log m ))
SEED queries if every x ∈ X lies on the grid over [ − 1 , 1] m with stepsize 2 − B/m .
It should be noted that, unlike most previous algorithms, all our algorithms do not need knowledge of γ .
Moreover, all the bounds above can be turned from expectation to high probability.
Finally, we show that the algorithms of Theorem 1 and 2 are nearly optimal:
Theorem 4.
For all m ≥ 2 , all k ≥ 2 , and all γ ≤ m − 3 / 2 / 16 there exists a distribution of instances with k classes in R m with strong convex hull margin γ where any randomized algorithm using SEED and LABEL queries that returns C with probability at least 12 makes at least (cid:4) k 2 (cid:5) m 24 log 12 γ total queries in expectation.
1.1 Preliminaries and notation
The input to our problem is a pair ( X, k ) , where X ⊂ R m and k ∈ N with 2 ≤ k ≤ n = | X | .
The algorithm has access to oracles O LABEL and O SEED which provide respectively LABEL and SEED queries.
The oracles O LABEL , O SEED behave consistently with some target classifier h :
X → [ k ] .
For any x ∈ X , LABEL ( x ) returns h ( x ) .
For any U ⊆ X and any i ∈ [ k ] , SEED ( U, i ) returns an abitrary element x ∈ U
∩ C i if U ∩ C i ̸
= ∅ , and NIL otherwise,
where C i = h − 1 ( i ) .
We often think of h as of the partition C =
( C 1 , . . .
, C k )
and we call each C i a class or cluster .
A pseudometric is a symmetric and subadditive function d : R m × R m → R ≥ 0 such that d ( x, x ) =for all x ∈ R m ; unlike a metric, d ( x, y ) can be 0 for x ̸ = y .
In this work d is always induced by a seminorm and thus homogeneous and invariant under translation: d ( u + ax, u + ay ) =
| a | d ( x, y ) for all x, y, u ∈ R m and all a ∈ R .
For a pseudometric d and a set A ⊂ R m , we let ϕ d ( A )
=
sup { d ( x, y ) : x, y ∈ A } denote the diameter of A under d .
For x ∈ R m and r ≥ 0 we denote by B md ( x, r ) and S m − 1 d ( x, r ) respectively the closed ball and the hypersphere with center x and radius r in R m under d .
When d is omitted we assume d = d euc where d euc is the Euclidean metric.
We may also omit the superscript if clear from the context.
The distortion of a (pseudometric) d is κ d = sup u,v ∈ S m − 1 (0 , 1)
∥ u ∥ d / ∥ v ∥ d .
For any set A ⊂ R m , any µ ∈ R m , and any λ > 0 , let σ ( A, µ, λ ) =
µ + λ ( A − µ ) be the scaling of A about µ by a factor of λ .
For two sets A, B ⊂ R m , we write A ≤ λB if A ⊆ σ ( B, z, λ ) for some z ∈ R m .
We may use x in place of A if A = { x } .
If A is bounded, then MVE( A ) denotes the minimum-volume enclosing ellipsoid (MVEE, or Löwner-John ellipsoid) of A .
Our proofs repeatedly use John’s theorem; that is, σ ( E, µ, 1 /m )
⊆ conv( A ) where µ is the center of E = MVE
( A ) and conv( A ) is the convex hull of A .
Given A, B ⊆
R m , we say that A and B are linearly separable with margin r if there exist u ∈ S m − 1 (0 , 1) and b ∈ R such that ⟨ u, x ⟩ + b ≤ − r for all x ∈ A and ⟨ u, x ⟩
+ b ≥ r for all x ∈ B .
We consider classifiers satisfying the following property:
Definition 5.
A class C i has strong convex hull margin γ > 0 if there exists a pseudometric d i induced by a seminorm over R m such that d
i (conv( C j ) , conv( C i )) > γ ϕ d
i ( C i ) for all j ∈ [ k ] \{ i } .
If this holds for all i ∈ [ k ] then we say C has strong convex hull margin γ .
Remarks.
The margin of Definition 5 captures natural scenarios that SVM margin does not.
For instance, suppose we are clustering fruits on the basis of weight and colour.
First, a fruit weighting more than, say, 1 . 5 times the typical weight of a species probably does not belong to it; but the typical weight varies greatly across species.
Our margin captures this scenario, as it is expressed as a fraction of the class’ diameter.
Second, different fruit species have different separating features; for instance, weight does not separate well oranges from bananas, but colour does.
Our margin captures this aspect, too, by allowing the metric that determines the margin to be a function the class.
It is also known that the SVM margin γ SVM can be arbitrarily smaller than γ ; for instance there are simple cases with γ > 1 but γ SVM <
e − n (see Bressan et al.
[2021a]).
Hence a large γ does not imply good bounds for standard algorithms based on SVM margin (e.g., the Perceptron).
Related work
It is well known that active learning may achieve exponential savings in label complexity.
That is, there are natural concept classes that can be learned with a number of LABEL queries exponentially smaller than that of passive learning.
Hanneke and Yang [2015] characterize the label complexity of concept classes in terms of their star number.
However, the star number of many natural classes such as linear classifiers is unbounded, implying a strong lower bound of Ω( n ) LABEL queries.
This and other negative results motivated research on enriched queries.
Kane et al.
[2017] prove that active learnability is characterized by the inference dimension of the concept class H under the set of allowed queries Q , as long as those queries are local (i.e., are a function of a constant number of instances).
This yields exponential savings when H is the class of linear separators and
Q contains label queries and comparison queries (which, given two points, reveal which one is closer to the decision boundary), provided the classes have SVM margin or bounded bit complexity.
Hopkins et al.
[2020] give similar results under distributional assumptions.
Unfortunately, bounded inference dimension does not automatically yield efficient algorithms, although it implies active learning algorithms with bounded memory [Hopkins et al., 2021].
SEED and their variants are motivated and used by Hanneke [2009] as positive example queries , by Balcan and Hanneke [2012] as conditional class queries , and by Beygelzimer et al.
Attenberg and Provost [2010] as search queries .
They are also used implicitly by Tong and Chang [2001], Doyle et al.
[2011], and Vikram and Dasgupta [2016].
SEED queries have been used in cluster recovery [Bressan et al., 2021b] and yield exponential savings in non-realizable learning settings [Balcan and Hanneke, 2012].
It also easy to see that SEED queries are equivalent to partial equivalence queries of Maass and Turán [1992] and to subset plus superset queries of Angluin [1988].
To the best of our knowledge, no work combines LABEL and SEED as we do here.
Little is known about the SEED complexity of learning a concept class H actively in polynomial time.
On the one hand, the inference dimension lower bounds of Kane et al.
[2017] are inapplicable, as
SEED queries are not local.
On the other hand the Littlestone dimension of H yields an upper bound, but not necessarily an efficient algorithm; in fact, it is well known that (some sub-problem solved by)
Halving is hard in general, see Gonen et al.
For k = 2 , we can use SEED to emulate equivalence queries, for which polynomial-time algorithms are known in some special cases.
In particular, the algorithm of Maass and Turán [1994] could replace our cutting-planes subroutine under an implicit discretization of the space through a grid with step-size O ( γ / m 4 ) .
However, this gives a polynomial-time algorithm that uses O ( m 2 log m / γ ) SEED queries, which is O ( m ) times our bound.
Moreover, Maass and Turán [1994] use proper equivalence queries (i.e., the queried concept must be in the class), for which they show a lower bound of Ω( m 2 log m / γ ) .
Finally, these techniques do not seem to extend to the case k > 2 .
Our notion of margin strengthens the convex hull margin of Bressan et al.
[2021a] by requiring d (conv( C j ) , conv( C i )) >
γϕ ( C i ) rather than d ( C j , C i ) > γϕ ( C i ) .
It is not hard to see that the convex hull margin can be arbitrarily smaller than our strong convex hull margin.
Finally, the polytope margin of Gottlieb et al.
[2018] assumes that each class is in the intersection of a finite number of halfspaces with margin.
It is easy to see that this condition is strictly stronger than ours.
Upper Bounds
This section gives the proofs of Theorem 1 and Theorem 2.
The algorithm behind both theorems has two phases which are described in the next subsections.
The case k > 2 is essentially the same as for k = 2 , except for an adaptation in the second phase.
3.1 The First Phase: Rounding the Classes
The first phase of our algorithms learns what we call an α -rounding of X .
Definition 6.
An α -rounding of X (w.r.t.
h ) is a sequence of pairs (( X i , E i ))
i ∈
[ k ] where ( X i ) i ∈
[ k ] is a partition of X , and where E i for i ∈ [ k ] is an ellipsoid such that X i ⊆
E i and E i ≤ α conv( C i ) .
The idea is that, if (( X i , E i )) i ∈
[ k ] is an α -rounding of X , then E i gives an approximation of the pseudometric d i witnessing the strong convex hull margin of C i .
Indeed, let p i be the pseudometric induced by E i , the one such that E i =
B p i ( µ i , 1) where µ i is the center of E i ; we prove:
Lemma 7.
If (( X i , E i ))
i ∈
[ k ] is an α -rounding of X then p
i (conv( X i ∩ C i ) , conv( X i ∩ C j )) ≥ γα for all distinct i, j ∈ [ k ] .
Proof.
If µ i is the center of E i , then E i =
B p i ( µ i , 1) .
Let d i be any pseudometric witnessing that
C i has strong convex hull margin γ > 0 .
As the margin is invariant under scaling, we can assume ϕ d
i ( C i ) = 1 and conv( C i ) ⊆
B d i ( z i , 1) for some z i ∈ R m .
Therefore:
[Equation]
As p i and d i are homogeneous and invariant under translation this implies p i ≥
d i α and thus p i (conv( X i ∩ C j ) , conv( X i ∩ C i )) ≥
α d i (conv( X i ∩ C j ) , conv( X i ∩ C i )) .
Moreover, by monotonicity under taking subsets and by the margin assumption d i (conv( X i ∩ C j ) , conv( X i ∩ C i )) ≥ d i (conv( C j ) , conv( C i )) ≥ γϕ d i ( C i ) = γ .
Combining the two inequalities yields the thesis.
We will use Lemma 7 in the second phase.
First, we show how to compute an α -rounding of X efficiently.
We sample points independently and uniformly at random from X until we find Θ( m 2 ) points S i with the same label i .
As the VC dimension of ellipsoids in R m is O ( m 2 ) , by standard generalization error bounds with constant probability the MVE of S i contains at least half of C i .
We then store that MVE together with the index i , remove S i from X , and repeat until X becomes empty.
At that point for each i ∈ [ k ] we “merge” together all points in the MVEs that were computed for class i , and compute the MVE of this merged set.
We show that this produces an α -rounding of X after O ( k log n ) rounds in expectation.
The resulting algorithm Round is listed below; Figuredepicts its behaviour on a toy example.
Lemma 8.
Round( X, k ) returns an m 2 ( m + 1) -rounding of X in time poly( n + m ) using
O ( k 2 m
2 log n ) LABEL queries in expectation.
Proof sketch.
First we show that E i ≤ m 2 ( m + 1) conv( C i ) for all i ∈ [ k ] .
This is trivial if E i = ∅ , so let E i ̸
= ∅ and let ℓ i ≥ 1 be the value of h
i at return time.
For every h = 1 , . . . , ℓ i let E hi =
MVE( S hi ) and let µ hi be the center of E hi .
Using John’s theorem one can show that σ (cid:0) E i , µ i , 1 m (cid:1) ⊆ conv (cid:83) ℓ i h
=1 σ (cid:0) conv( S hi ) , µ hi , m (cid:1) and σ (cid:0) conv( S hi ) , µ hi , m (cid:1) ⊆
σ (conv( C i ) , µ, m ( m + 1)) .
By taking the union over all h ∈ [ ℓ i ] we conclude that σ (cid:0) E i , µ i , 1 m (cid:1) ⊆
σ (conv( C i ) , µ, m ( m + 1))
, that is, E i ≤ m 2 ( m +1) conv( C i ) .
It is also easy so see that ( X i ) i ∈
[ k ] is a partition of X , hence (( X i , E i ))
i ∈
[ k ] is an m 2 ( m +1)
-rounding of X .
For the running time, the for loops perform k ≤ n iterations, and the while loop performs at most n iterations as each iteration strictly decreases the size of X .
The running time of any iteration is dominated by the computation of MVE( S i ) or MVE( X i ) , which takes time poly( n + m ) , see above.
Hence Round( X, k ) runs in time poly( n + m ) .
For the query bounds, the while loop makes
O ( m 2 k ) LABEL queries per iteration.
By standard generalization bounds, since the VC dimension of
Algorithm for i ∈ [ k ] do h i ←while X ̸ = ∅ do draw points independently u.a.r.
from X and LABEL them until for some i ∈ [ k ] we draw a (multi)set of cm 2 points from C i h
i ← h
i +
S h
i i ← the sample of cm 2 points from C
i
X h
i
i ← X ∩ MVE( S h
i i )
X ← X \ X h
i i for i ∈
[ k ] do
X i ← X 1 i ∪ . . .
∪ X h
i
i (set to ∅ if h i = 0 )
E i ← MVE( X i ) (set to ∅ if X i = ∅ ) return (( X i , E i ))
i ∈
[ k ]
[Figure]
ellipsoids in R m is O ( m 2 ) , E hi contains at least half of X ∩ C i with probability at least 12 , and thus the expected number of rounds before X becomes empty is in O ( k lg n ) , see Bressan et al.
[2021a].
We conclude that Round( X, k ) uses O ( m 2 k 2 lg n ) LABEL queries in expectation.
3.2 The Second Phase: Finding a Separator via Cutting Planes
Let (( X i , E i ))
i ∈
[ k ] be the output of Round( X, k ) , and fix i ∈ [ k ] .
For each j ∈ [ k ] \ { i } , we want to separate X i ∩
C
i from X i ∩ C j .
To this end, first we use E i to perform a change of coordinates; this puts X i inside the unit ball and ensures that X i ∩ C i and X i ∩ C j are linearly separated with margin γ SVM =
Ω( γm − 3 ) .
Next, by calling C i the positive class ( +1 ) and C j the negative class ( − 1 ), and letting X = X i for simplicity, one can reduce the task to the following problem.
Consider a partial classifier h :
X → { +1 , − 1 , ∗} .
The algorithm has access to an oracle answering queries
SEED ( U, y ) where U ⊆ X and y ∈ { +1 , − 1 } , and its goal is to compute a separator of X :
Definition 9.
Let X ⊂ R m and h : X → { +1 , − 1 , ∗} .
A separator of X (w.r.t.
h ) is a partition ( X + , X − ) of X such that, for every x ∈ X , if h ( x ) =
+1 then x ∈ X
+ and if h ( x ) = − 1 then x ∈ X − .
A separator of X can be learned, for instance, by the Perceptron (using SEED to find counterexamples).
However, this would yield a query and running time bound of O (1 /γ 2 SVM )
=
O ( m 6 /γ 2 ) .
We provide
CPLearn , a cutting-plane algorithm based on SEED that is much more query-efficient (in fact, near-optimal):
Theorem 10.
Let X ⊂ R m and h : X → { +1 , − 1 , ∗} , and suppose h − 1 (+1) and h − 1 ( − 1) are linearly separable with margin r .
Given X and access to SEED for labels { +1 , − 1 } , CPLearn( X ) computes a separator of X w.r.t.
h using O ( m log Rr ) SEED queries in expectation, where R = max x ∈ X ∥ x ∥ 2 , and running with high probability 6 in time poly( m + | X | ) .
Proof.
( Sketch )
First, we lift X to R m +1 .
This reduces the problem to finding a homogeneous linear separator.
To this end we let X ′ =
{ x ′ : x ∈ X } where x ′ is obtained by appending to x an ( m +1) -th coordinate that is equal to R , and we extend h to X ′ in the obvious way.
It is easy to prove that X ′ has radius at most 2 R and that in X ′ the two classes are linearly separable with margin r 2 .
Next, we learn a separator of X ′ w.r.t.
h via cutting planes—see, e.g., Mitchell [2003].
Let V 0 =
B m +
Every point u ∈ V 0 identifies the halfspace H ( u )
= { z ∈ R m +1 : ⟨ u, z ⟩ ≥ 0 } .
For i = 1 , 2 , . . .
, V i will be our version space, and we compute V i +1 from V i as follows.
Let µ i be the center of mass of V i , and let X ′ i
=
X ′
∩ H ( µ i ) .
By issuing SEED ( X ′ i , − 1) and SEED ( X ′ \ X
′ i , +1)
we learn whether ( X ′ i , X
′ \ X
′ i ) is a separator of X ′ w.r.t.
h , in which case we return the corresponding partition of X , or we obtain a point u i .
In the second case, we let V i +
= V i ∩ U
i where U
i = { x ∈ R m +1 :
h ( u i ) · ⟨ u i , x ⟩ ≥ 0 } .
By [Gilad-Bachrach et al., 2004, Theorem 2] this procedure returns a separator of X ′ w.r.t.
h using at most
[Equation]
Unfortunately, computing µ i is hard in general [Rademacher, 2007].
We instead compute an estimate ˆ µ i that, used in place of µ i , ensures vol( V i +1 )
vol
( V i ) is bounded away from 1 with high probability; the expected query bound follows by adapting the proof of [Gilad-Bachrach et al., 2004].
Assume for the moment that V i is well-rounded—that is, it contains a ball of radius r = poly( m ) and is contained in a ball of radius 1 .
To compute ˆ µ i we average over poly( n + m ) independent uniform points from
V i , which can be draw efficiently thanks to the rounding condition.
At this point we use ˆ µ i in place of µ i to invoke SEED and obtain a violated constraint U i .
Howewer, setting V i
+
= V i ∩ U
i could make V
i +1 far from rounded (too “thin”), making sampling inefficient at the next round.
Therefore we rotate U i so to obtain a weaker constraint U ∗ i , one that still contains V i ∩
U
i
but that has ˆ µ i on its boundary, and let V i +
= V i ∩ U ∗ i .
By the assumption on ˆ µ
i
this implies that vol( V i +1 ) ≥ 13 vol( V i ) ; therefore by sampling uniform points from V i we can obtain a large sample in V
i +1 ,
from which we can put V
i +1 in a rounding position.
See the full proof for all the details.
To the best of our knowledge, CPLearn is the first efficient algorithm that achieves the query upper bound of Theorem 10, even for the special case of SVM margin.
3.3 Wrap-Up
We wrap up our algorithms, starting with the case k = 2 ; the case k ≥ 2 is slightly more involved.
Algorithm 2: BinLearn( X ) (( X 1 , E 1 ) , ( X 2 , E 2 ))
← Round( X ) for i ← 1 , 2 do change system of coordinates so that E i becomes the unit ball ( X i + , X i − ) ← CPLearn( X i ) with h : X i → { 1 , 2 } return ( X 1+ ∪ X 2 − , X 2+ ∪ X 1 − )
Theorem 11.
Suppose k = 2 .
Then BinLearn( X ) returns C
= ( C 1 , C 2 ) in time poly( n + m ) using in expectation O ( m 2 log n ) LABEL queries and O ( m log mγ )
SEED queries.
Proof.
By Lemma 8, Round( X ) runs in time poly( n + m ) , makes O ( m 2 log n ) LABEL queries in expectation, and returns an O ( m 3 ) -rounding of X .
It is immediate to see that, after the change of coordinates, X i has radius R ≤ 1 , while C 1 ∩ X 1 and C
∩ X 1 are separated linearly with margin r = Ω( γm − 3 ) .
By Theorem 10 then, CPLearn( X i ) returns the partition of X i induced by h in time poly( | X i | + m )
= poly( n + m ) using O (cid:0)
m log Rr (cid:1) =
O (cid:0) m log mγ (cid:1) expected SEED queries.
For k ≥ 2 we proceed as follows.
Let k =
[ k ] .
We take X i for each i ∈ k in turn, and for each j ∈ k \ i , we use CPLearn to compute a separator for i, j in X i .
By intersecting the left side of all those separators we obtain X i ∩ C i .
Then we recurse on X i \ C i , updating k to k \ i .
The resulting algorithm KClassLearn is listed below and yields:
Theorem 12.
KClassLearn( X, [ k ]) returns C in time poly( n + m ) using in expectation
O ( k !
k
2 m 2 log n ) LABEL queries and O (cid:0) k !
k 2 m log mγ (cid:1) SEED queries.
Proof.
We adapt the proof of Theorem 11.
Observe that KClassLearn( X, [ k ]) makes at most min( k ! , n ) recursive calls; the n in the min comes from the fact that any given (recursive) call learns the label of at least one unlabeled point.
Now, every (recursive) call makes one invocation to Round( X ) , which by Lemma 8 uses time poly( n + m ) and O ( k 2 m 2 log n ) LABEL queries, and
O ( k 2 ) invocations to CPLearn( X i ) , each of which by Theorem 10 uses poly( n + m ) time and
O (cid:0) m log mγ (cid:1) SEED queries.
k ← | k | if k = 1 then query any point of X and label all of X accordingly else (( X i , E i ))
i ∈
[ k ] ← Round( X ) for i ∈ k do change system of coordinates so that E i becomes the unit ball for j ∈ k \ i do ( C ij , C ij )
← CPLearn( X i ) with h : X i → { i, j } (cid:98) C i ← (cid:84) j ∈ k \ i C ij mark all of (cid:98) C i with label i if X i \ (cid:98)
C i ̸
= ∅ then KClassLearn( X i \ (cid:98) C i , k \ i )
Lower Bounds
This section gives a detailed sketch of the proof of Theorem 4, recalled here for convenience:
Theorem 4.
For all m ≥ 2 , all k ≥ 2 , and all γ ≤ m − 3 / 2 / 16 there exists a distribution of instances with k classes in R m with strong convex hull margin γ where any randomized algorithm using SEED and LABEL queries that returns C with probability at least 12 makes at least (cid:4) k 2 (cid:5) m 24 log 12 γ total queries in expectation.
We first give the sketch for k = 2 , and then extend it to k ≥ 2 .
For a full proof see Appendix B. Setup.
The construction is adapted from Proposition 2 of Thiessen and Gärtner [2021].
Let e 1 , . .
, e m be the canonical basis of R m and let ℓ =
⌊ 1 / √ 2 γ √ m ⌋ ; note that γ ≤ m − 3 / 2and m ≥ 2 ensure ℓ ≥ 4 .
Let p = m − 1 , and for each i ∈ [ p ] and j ∈ [ ℓ ] define x ji = e i + j · e m .
Finally, let X =
{ x ji : i ∈ [ p ] , j ∈ [ ℓ ] } and define the concept class H =
(cid:110)(cid:83) i ∈
[ p ] { x 1 i , . . .
, x ℓ i i } : ( ℓ 1 , . . . , ℓ p ) ∈ [ ℓ ] p (cid:111) .
Let
C =
( C 1 , C 2 ) be any partition of X such that C 1 ∈ H .
One can easily verify that C has strong convex hull margin
1 2 ℓ 2 √ m ≥ γ .
See
Figure
for
reference.
Query bound.
Let V
= { ( C 1 , C 2 ) :
C 1 ∈ H} .
This is the initial version space.
We let the target concept C =
( C 1 , C 2 ) be drawn uniformly at random from V 0 .
Note that for k = 2 , any lower bound on the number of SEED queries alone, also holds for any combination of SEED and LABEL queries, as
LABEL ( x ) can be simulated by SEED ( x, 1) .
Thus, without loss of generality, we can assume that the algorithm is only using SEED queries.
For all t = 0 , 1 , . . .
, we denote by V t the version space after the first t SEED queries made by the algorithm.
Now fix any t ≥ 1 and let SEED ( U, y ) be the t -th such query.
Without loss of generality we assume y = 1 ; a symmetric argument applies to y = 2 .
If
U ∩ C 1 contains a point x whose label can be inferred from the first t − 1 queries, then we return
S+ + + + + − − − − − S+ + − − − − − − − −
[Figure]
x .
Therefore we can continue under the assumption that U does not contain any such point (doing otherwise cannot reduce the probability that the algorithm learns nothing).
The oracle answers so to maximize | V t | | V t − 1 | , as described below.
For each i ∈ [ p ] let S i
= { x ji : j ∈ [ ℓ ] } .
We consider S i as sorted by the index j .
Let Z i be the subset of S i in the disagreement region of V t − 1 together with the point in S
i preceding this region; observe that this point always exists, as x 1 i ∈ C 1 is in the agreement region.
Note that Z i is necessarily an interval of S i .
We let U i
= Z i ∩ U for each i ∈ [ p ] and P ( U ) =
{ i ∈ [ p ] : U i ̸ = ∅} .
For every i ∈ P ( U ) , we let α i be the fraction of points of Z i that precede the first point in U i .
Let x ∗ i = arg max { j : x ji ∈ S i ∩ C 1 } .
Observe that | V t − 1 | = (cid:81) i ∈
[ p ] | Z i | .Indeed, x ∗ i is uniformly distributed over Z i ; either x ∗ i is a point in the disagreement region of S i , or the disagreement region of S i is fully contained in C 2 and x ∗ i is the point preceding the disagreement region of S i .
Now we show that E [ | V t − 1 | / | V t | ] ≤ m .
Let E be the event that SEED ( U, 1)
=
NIL .
Write:
[Equation]
We bound each one of the two terms in the right-hand side.
For the first term, note that E holds if and only if U i ∩ C 1 = ∅ for all i ∈ P ( U ) .
Since x ∗ i is uniformly distributed over Z i , for all i ∈ P ( U ) we have Pr( C 1 ∩ U i = ∅ )
=
α i , and since the distributions of those points are independent, then Pr( E ) =
(cid:81)
i ∈ P ( U ) α i .
If Pr( E ) > 0 and E holds, then x ∗ i is uniformly distributed over the first α i
| Z i | points of Z i , as the rest of Z i belongs to C 2 .
This holds independently for all i , thus:
[Equation]
It follows that Pr( E ) E (cid:104)
[Equation]
E (cid:105) ≤ 1 .
Let us turn to the second term.
If E does not hold, then SEED ( U, 1) returns
the smallest point x ∈ U i for any i ∈ P ( U ) such that C 1 ∩
U i ̸
= ∅ (note that necessarily x ∈ C 1 ).
For any fixed i ∈ P ( U ) , the probability of returning the smallest point of U i is bounded by Pr( C 1 ∩ U i ̸ = ∅ ) , which is 1 − α i ; and if this is the case, then we have | V t | =
(1 − α i ) | V t − 1 | .
Thus:
[Equation]
So the two terms of (2) are both bounded by 1 ; we conclude that E (cid:104) | V t − 1 | | V t | (cid:105) ≤ 2 .
Next, fix any ¯ t ≥ 1 and let log = log 2 .
By the concavity of log and by Jensen’s inequality:
[Equation]
Since E (cid:104) | V t − 1 | | V t | (cid:105) ≤ 2 , the right-hand side is at most ¯ t .
Now, since | V 0 | =
ℓ p = ℓ m − 1 , by Markov’s inequality, and since ( m − 1) log ℓ − log 2 ≥ ( m − 1)log ℓ
[Equation]
m log ℓ:
[Equation]
( |
[Equation]
=
[Equation]
Now let T be the random variable counting the number of queries spent by the algorithm, and let V T be the version space at return time.
Since C is uniform over V T and C is returned with probability at least 12 , then Pr( | V T | ≤ 2) ≥ 12 .
By (6) and linearity of expectation,
[Equation]
Therefore E [ T ] ≥ m log ℓ. Now, since ℓ ≥ 4 then ℓ ≥ 4 5 √ 2 γ √ m , which since m ≤ (16 γ ) − 2 / 3 yields, after calculations, ℓ ≥ 3 (cid:112) 1 / γ · 4 4 / 3 5 √ 2 > 0 . 89 3 (cid:112) 1 / γ .
This shows that E [ T ] > m 24 log 12 γ , concluding the proof for k = 2 .
Extension to k ≥ 2.
For each s ∈ (cid:4) k 2 (cid:5) and each pair of classes C 2 s − 1 , C 2 s , use the construction above shifted along the m -th dimension by ( s − 1) ℓ .
One can easily verify that learning C is as hard as learning (cid:4) k 2 (cid:5) independent binary classifiers, for each of which the bound above holds.
Conclusions and Future Work
We have shown that, with a careful combination of LABEL and SEED queries, one can overcome the limitations of each query alone and get the “best of both worlds”: an algorithm that achieves exponential savings and, simultaneously, has running time polynomial in the dimension of the space.
Our work leaves open a few problems.
The first problem is to understand the tradeoff between the two query types: how many LABEL does one need if one is allowed only Q SEED ?
The second problem is whether, for the one-sided case, one can achieve a query rate that is independent of the distortion κ d , as we did for the multiclass case.
The third problem is whether one can improve the dependence of our bounds on the number k of classes, ideally bounding it by a polynomial.
Acknowledgments and Disclosure of Funding
The authors gratefully acknowledge partial support by the Google Focused Award “Algorithms and Learning for AI” (ALL4AI).
Nicolò Cesa-Bianchi is also supported by the MIUR PRIN grant
Algorithms, Games, and Digital Markets (ALGADIMAR) and by the EU Horizon 2020 ICTresearch and innovation action under grant agreement 951847, project ELISE (European Learning and Intelligent Systems Excellence).