Active Learning Polynomial Threshold Functions
Abstract
We initiate the study of active learning polynomial threshold functions (PTFs).
While traditional lower bounds imply that even univariate quadratics cannot be non-trivially actively learned, we show that allowing the learner basic access to the derivatives of the underlying classifier circumvents this issue and leads to a computationally efficient algorithm for active learning degree- d univariate PTFs in ˜
O ( d 3 log(1 /εδ )) queries.
We extend this result to the batch active setting, providing a smooth transition between query complexity and rounds of adaptivity, and also provide near-optimal algorithms for active learning PTFs in several average case settings.
Finally, we prove that access to derivatives is insufficient for active learning multivariate PTFs, even those of just two variables.
Introduction
Today’s deep neural networks perform incredible feats when provided sufficient training data.
Sadly, annotating enough raw data to train your favorite classifier can often be prohibitively expensive, especially in important scenarios like computer-assisted medical diagnoses where labeling requires the advice of human experts.
This issue has led to a surge of interest in active learning , a paradigm introduced to mitigate extravagant labeling costs.
Active learning, originally studied by Angluin in 1988 [1], is in essence formed around two basic hypotheses: raw (unlabeled) data is cheap and not all data is equally useful.
The idea is that by adaptively selecting only the most informative data to label, we can get the same accuracy without the prohibitive cost.
As a basic example, consider the class of thresholds in one dimension.
Identifying the threshold within some ε accuracy requires about 1 /ε labeled data points, but if we are allowed to adaptively select points we can use binary search to recover the same error in only log(1 /ε ) labels, an exponential improvement!
Unfortunately, there’s a well-known problem with this approach: it breaks down for most non-trivial classifiers beyond 1 D-thresholds [2], providing no asymptotic benefit over standard non-adaptive methods.
This has lead researchers in recent years to develop a slew of new strategies overcoming this obstacle.
We follow an approach pioneered by Kane, Lovett, Moran, and Zhang (KLMZ) [3]: asking more informative questions.
KLMZ suggest that if we are modeling access to a human expert, there’s no reason to restrict ourselves to asking only about the labels of raw data; rather, we should be allowed access to other natural application-dependent questions as well.
They pay particular attention to learning halfspaces in this model via “comparison queries,” which given x, x ′ ∈ R d ask which point is closer to the bounding hyperplane (think of asking a doctor “which patient is more sick?”).
Such queries had already shown promise in practice [4–6], and KLMZ proved they could be used to efficiently active learn halfspaces in two-dimensions, recovering the exponential improvement seen for 1 D-thresholds via binary search.
Beyond two dimensions, however, all known techniques either require strong structural assumptions [3, 7], or the introduction of complicated queries [8, 9] requiring infinite precision, a significant limitation in both theory and practice.
The study of active learning halfspaces can be naturally viewed as an attempt to extend the classical active learning of 1 D-thresholds to higher dimensions .
In this work, we take a different approach and instead study the generalization of this problem to higher degrees .
In particular, we initiate the study of active learning polynomial threshold functions , classifiers of the form sign ( p ( x )) for x ∈ R and p some underlying univariate polynomial.
When the degree of p is 1 , this reduces to the class of 1 D-thresholds.
Similar to halfspaces, standard arguments show that even degree-two univariate PTFs cannot be actively learned.
1 To this end, we introduce derivative queries , a natural class-specific query-type that allows the learner weak access to the derivatives of the underlying PTF p .
Derivative queries are well-motivated both in theory and practice.
A simple example is the medical setting, where a first-order derivative might correspond to asking “Is patient X recovering, or getting sicker?”
Derivatives also play an essential role in our sensory perception of the world.
Having two eyes grants us depth perception [10], allowing us to compute low-order derivatives across time-stamps to predict future object positions (e.g. for hunting, collision-avoidance).
Multi-viewpoint settings also allow access to low order derivatives by comparing nearby points; one intriguing example is the remarkable sensory echolocation system of bats, which emit ultrasonic waves while moving to learn the structure of their environment [11].
While high order derivatives may be more difficult to compute for a human (or animal) oracle, they still have natural implications in settings such as experimental design where queries are measured mechanically (e.g. automated tests of a self-driving car system might reasonably measure higher order derivatives of positional data).
Such techniques have already seen practical success with other query types typically considered too difficult for human annotators (see e.g. the survey of Sverchkov and Craven [12] on automated design in biology).
Our main result can be viewed as theoretical confirmation that this type of question is indeed useful: derivative queries are necessary and sufficient for active learning univariate PTFs .
We prove that if the learner is allowed access to sign ( p ( i ) ( x )) (the i -th order derivative of p ), PTFs are learnable in O (log(1 /ε )) queries, but require Ω(1 /ε ) queries if the learner is missing access even to a single relevant derivative.
We generalize this upper bound to the batch setting as well, giving a smooth interpolation between query complexity and rounds of communication with data annotators (which have costly overhead in practice).
We also study active learning PTFs beyond the worst-case setting.
Specifically, we consider a setup in which the learner is promised that both points in R and the underlying polynomial are drawn from known distributions.
We propose a general algorithm for active learning PTFs in this model based on coupon collecting and binary search, and analyze its query complexity across a few natural settings.
Notably, our algorithm in this model avoids the use of derivatives altogether, making it better adapted to scenarios like learning natural imagery where we expect the underlying distributions to be nice, but may not have access to higher order information like derivatives.
Finally, we note that all of our upper bounds (in both worst and average-case settings) actually hold for the stronger ‘perfect’ learning model in which the learner aims to query-efficiently label a fixed ‘pool’ of data with zero error.
Perfect learning is equivalent to active learning in the worst-case model [13, 3], but is likely harder in the average-case and requires new insight over standard techniques in our setting.
We end our work with a preliminary analysis of active learning multivariate PTFs, where we prove a strong lower bound showing access to derivative information is actually insufficient to active learn even degree-two PTFs in two variables.
We leave upper bounds in this challenging regime (e.g. through distributional assumptions, additional enriched queries) as a direction of future research.
1.1 Background
We briefly overview the basic theory of PAC-learning (in both the “passive” and “active” settings) and of the main model we study, perfect learning .
We cover these topics in much greater detail in the supplementary materials.
PAC-learning, originally introduced by Valiant [14] and Vapnik and
Chervonenkis [15], provides a framework for studying the learnability of pairs ( X, H ) where X is a set and H
= { h : X → {− 1 , 1 }} is a family of binary classifiers.
A class ( X, H ) is said to be
PAC-learnable in n = n ( ε, δ ) samples if for all ε, δ > 0 , there exists an algorithm A which for all distributions D over X and classifiers h ∈ H , intakes a labeled sample of size n and outputs a good hypothesis with high probability:
[Equation]
where err D,h ( A ( S, h ( S )))
=
P x ∼ D
[ A ( S, h ( S ))
( x ) ̸ = h ( x )] .
Active learning is a modification of the PAC-paradigm where the learner instead draws unlabeled samples, and may choose whether or not they wish to ask for the label of any given point.
The goal is to minimize the query complexity q ( ε, δ ) , which measures the number of queries required to attain the same accuracy guarantees as the standard “passive” PAC-model described above.
In the batch setting, the learner may send points to the oracle in batches.
This incurs the same query cost as in the standard setting (a batch of m points costs m queries), but allows for a finer-grained analysis of adaptivity through the round complexity r ( ε, δ ) which measures the total number of batches sent to the oracle.
In this work, we study a challenging variant of active learning called perfect learning (variants of which go by many names in the literature, e.g. RPU-learning [16], perfect selective classification [13], and confident learning [3]).
In this model, the learner is asked to label an adversarially selected size- n sample from X .
The query complexity q ( n ) (respectively round complexity r ( n ) ) is the expected number of queries (respectively rounds) required to infer the labels of all n points in the sample.
Perfect learning is well known to be equivalent to active learning up to small factors in query complexity in worst-case settings, and is at least as hard as the latter in the average-case [13, 3].
We discuss these connections in more depth in Section 2.
In this work, we study the learnability of ( R , H d ) , the class of degree (at most) d univariate PTFs.
In the worst-case setting, we allow the learner access to derivative queries : for any x ∈ R in the learner’s sample, they may query sign ( f ( i ) ( x )) for any i = 0 , . . .
, d − 1 , where f ( i ) is the i -th derivative of f and f (0) is f itself.
1.2 Results
Our main result is that univariate PTFs can be computationally and query-efficiently learned in the perfect model via derivative queries.
Theorem 1.1 (Perfect Learning PTFs) .
The query complexity of perfect learning ( R , H d ) with derivative queries is:
[Equation]
Furthermore, there is an algorithm achieving this upper bound that runs in time ˜ O ( nd ) .
By standard connections with active learning, this implies PTFs are active learnable in query complexity Ω( d log(1 /ε ))
≤ q ( ε, δ ) ≤ ˜ O
(cid:0) d 3 log( 1 εδ ) (cid:1) when the learner has access to derivative queries.
Theorem 1.1 is based on a deterministic algorithm that iteratively learns each derivative given higher order information.
This technique necessarily requires a large amount of adaptivity which can be costly in practice.
To mitigate this issue, we give a randomized algorithm that extends Theorem 1.1 to the batch setting, providing a smooth trade-off between (expected) query-optimality and adaptivity.
Theorem 1.2 (Perfect Learning PTFs Batch Setting) .
For any n ∈ N and α ∈ (1 / log( n ) , 1] , there exists a randomized algorithm perfectly learning size n subsets of ( R , H d ) in [Equation] n
[Equation]
expected queries, and
[Equation]
expected rounds of adaptivity.
Moreover, the algorithm can be implemented in ˜ O ( n ) expected time.
When α = O (1 / log( n )) , this recovers the query complexity of Theorem 1.1 in expectation, but also gives a much broader range of options, e.g. sub-linear query algorithms in O (1) rounds of communication.
In fact it is worth noting that even in the former regime the algorithm uses only
O (log( n )) total rounds of communication, independent of the underlying PTF’s degree.
Finally, note that run-time is also near-optimal since Ω( n ) time is required even to read the input.
To complement these upper bounds, we also show that PTFs cannot be actively learned at all if the learner is missing access to any derivative.
Theorem 1.3 (Perfect Learning PTFs Requires Derivatives) .
Any learner using label and derivative queries that is missing access to f ( i ) for some 1 ≤ i ≤ d − 1 must make at least q ( n ) ≥ Ω( n ) queries to perfectly learn ( R , H d ) .
This implies the query complexity of active learning PTFs with any missing derivative is Ω(1 /ε ) .
In some practical scenarios, our worst-case assumption over the choice of distribution over R and
PTF h ∈ H
d may be unrealistically adversarial.
To this end, we also study a natural average case model for perfect learning, where the sample S ⊂ R and PTF
h ∈ H
d are promised to come from known distributions.
In Section 4, we show derivates are often unecessary in this regime by providing a generic label-only algorithm and proving query efficiency in basic scenarios.
This is better suited than our worst-case analysis to practical scenarios like learning natural 3D-imagery, where we expect objects to come from nice distributions but don’t necessarily have higher order information.
We start by considering the basic scenario where both the sample and roots of our PTF are drawn uniformly at random from the interval [0 , 1] , a distribution we denote by U [0 , 1] .
Theorem 1.4 (Learning PTFs with Uniformly Random Roots) .
The query complexity of perfect learning ( R , H d ) when promised that the sample and roots are chosen from U [0 , 1] is:
[Equation]
While studying the uniform distribution is appealing due to its simplicity, similar results can be proved for somewhat more realistic distributions.
As an example, we study the case where the (intervals between) roots of our polynomial are drawn from a Dirichlet distribution Dir ( α ) , which has pdf:
[Equation]
where x i ≥ 0 and (cid:80) x i = 1 .
This generalizes drawing a uniformly random point on the d -simplex.
Theorem 1.5 (Learning PTFs with Dirichlet Roots) .
The query complexity of perfect learning ( R , H d ) when the subsample S ∼ U [0 , 1] and h ∼ Dir ( α ) is at most
[Equation]
for
[Equation]
=
[Equation]
for α ≥ 2 , and
[Equation]
for α ≥
[Equation]
Moreover, this result is tight for constant α and sufficiently large n .
So far we have only discussed univariate PTFs.
One might reasonably wonder to what extent our results hold for multivariate PTFs.
In fact, we show that derivative queries are insufficient (in the worst-case setting) for learning PTFs of even two variables.
Theorem 1.6 (Derivatives Can’t Learn Multivariate PTFs) .
Let ( R 2 , H 22 ) denote the class of degreetwo, two-variate PTFs.
The query complexity of perfectly learning ( R 2 , H 22 ) is q [Equation] n ) , even when the learner may query the sign of the gradient and hessian on any point in its sample.
In other words, multivariate PTFs cannot be actively learned via access to basic derivative queries in the worst-case.
It remains an interesting open problem whether there exist natural query sets that can learn multivariate PTFs, or whether this issue can be avoided in average-case settings; we leave these questions to future work.
1.3 Related work
Active Learning Halfspaces: While to our knowledge active learning polynomial threshold functions has not been studied in the literature, the closely related problem of learning halfspaces is perhaps one of the best-studied problems in the field, and indeed in learning theory in general.
It has long been known that halfspaces cannot be active learned in the standard model [2], but several series of works have gotten around this fact either by restricting the adversary, or empowering the learner.
The first of these two methods generally involves forcing the learner to choose a nice marginal distribution over the data, e.g. over the unit sphere [18], unit ball [19], log-concave [20], or more generally s -concave distributions [21].
The second approach usually involves allowing the learner to ask some type of additional questions.
This encompasses not only KLMZ’s [3] notion of enriched queries, but also the original “Membership query” model of Angluin [22] who allowed the learner to query any point in the overall instance space X rather than just on the subsample S ⊂ X .
This model is also particularly well-studied for halfspaces where it is called the point-location problem [17, 23–25, 7, 9], and was actually studied originally by Meyer auf der Heide [17] in the perfect learning model even before Angluin’s introduction of active learning.
Bounded degree PTFs may be viewed as a special set of halfspaces via the natural embedding to { 1 , x, x 2 , . . . } .
Given this fact, it is reasonable to ask why our work is not superseded by these prior methods for learning halfspaces.
The answer lies in the fact that the query types used in these works are generally very complicated and require infinite precision.
For instance, many use arbitrary membership queries (which are known to behave poorly in practice [26]), and even those that sacrifice on query complexity for simpler queries still require arbitrary precision (e.g. the “generalized comparisons” of [8]).
Indeed, learning halfspaces even in three dimensions with a simple query set remains an interesting open problem, and our work can be viewed as partial progress in this direction for sets of points that lie on an embedded low-degree univariate polynomial.
For instance, one could learn the set S
= { ( x, 3 x 5 , 5 x 7 ) : x ∈ [ n ] } ⊂ R 3 with respect to any underlying halfspace sign ( ⟨ v, ·⟩ + b ) in O (log n ) queries using access to standard labels and the derivatives of the underlying polynomial.
Active Learning with Enriched Queries: Our work also fits into a long line of recent studies on learning with enriched queries in theory and in practice.
As previously mentioned, Angluin’s [22] original membership query model can in a sense be viewed as the seminal work in this direction, and many types of problem-specific enriched queries such as comparisons [27, 4–6, 3, 8, 7, 28, 29, 9, 30], cluster-queries [31–39], mistake queries [40], separation queries [41], and more have been studied since.
Along with providing exponential improvements in query complexity in theory, many of these query types have also found use in practice [4, 5, 42, 43, 12].
Indeed even complicated queries such as Angluin’s original model that cannot be accurately assessed by humans [26] have found significant use in application to automated experimental design, where the relevant oracle is given by precise scientific measurements rather than a human (see e.g. the seminal work of King et al.
“The
Automation of Science” [43]).
While we view first or second order derivatives as reasonable query types for human experts, higher order derivatives are likely more useful in this latter setting, e.g. in application to dynamical systems where one tracks object movement with physical sensors.
Average Case Active Learning:
The average-case model we study in this work is the ‘perfect’ or ‘zero-error’ variant of the average-case active learning model introduced by Dasgupta [2] (and implicitly in earlier work of Kosaraju, Przytycka, and Borgstrom [44]).
These works gave a generic greedy algorithm for active learning finite concept classes ( X, H ) over arbitrary prior distributions whose query complexity is optimal to within a factor of O (log( | H | )) .
The exact constants of this approximation were later optimized in the literature on submodular optimization [45], and more recently extended to the batch setting [46].
These works differ substantially from our setting as they focus on giving a generic algorithm for average-case active learning, rather than giving query complexity bounds for any specific class.
Perhaps more similar to our general approach are active learning methods based on Hanneke’s disagreement coefficient [47], and Balcan, Hanneke, and Wortman’s [48] work on active learning rates over fixed instead of worst-case hypotheses.
Analysis based on these approaches typically takes advantage of the fact that for a fixed distribution and classifier, the minimum measure of any interval can be considered constant.
Our average-case setting can be thought of as a strengthening of this approach in two ways: first we are only promised (weak) concentration bounds on the probability this measure is small, and second we work in the harder perfect learning model.
This latter fact is largely what separates our analysis, as naive attempts at combining prior techniques with concentration lead to ‘imperfect’ algorithms (ones with a small probability of error).
Moving from the low-error to zero-error regime is in general a difficult problem, 3 but is important in high-risk applications like medical diagnoses.
4 Fixing this issue requires analysis of a new ‘capped’ variant of the coupon collector problem, and proving optimal query bounds requires further involved calculation that would be unnecessary in the low-error active regime.
Preliminaries
We now cover basic background on learning with enriched queries before sketching the proofs of our main results.
Detailed information on all background and full versions of all proofs can be found in the supplementary materials.
Learning with Enriched Queries: Recall our learner is allowed to make derivative queries , that is given S ⊂ R , the learner may query sign ( f ( i ) ( x )) for any x ∈ S, 0 ≤ i ≤ d − 1 .
Such queries have a number of natural interpretations, e.g. the relative distance of objects in image recognition (“is the pedestrian getting closer, or further away?”).
Given a PTF f ∈ H d and point x ∈ S , it is useful to consider the collection of all derivative queries on x which we call its sign pattern .
Definition 2.1 (Sign Pattern) .
The sign pattern of x ∈ R with respect to f ∈ H d is the vector in {− 1 , 1 } d +1 :
[Equation]
[Equation]
More generally, given a family of binary queries Q (e.g. labels and derivative queries), let Q h ( T ) denote the set of all possible query responses to x ∈ T given h (when Q consists of derivative queries, Q h ( T ) is the set of sign patterns in T ).
Since we can rule out any hypotheses h ′ ∈
H such that Q h ′ ( T )
̸
= Q h ( T ) , we will be interested in the set of consistent hypotheses, H | Q h ( T ) , which satisfy Q h ′ ( T ) =
Q h ( T ) .
We say that Q h ( S ) infers the label of a point x ∈ X when x only has one possible label under the set of consistent hypotheses, that is when for some z ∈ {− 1 , 1 } : ∀ h ′ ∈ H | Q h ( S ) : sign ( h ′ )
( x ) =
z.
Inference Dimension: In their seminal work on the enriched query model, KLMZ [3] introduced inference dimension , a combinatorial parameter that exactly characterizes the query complexity of both perfect and active learning under enriched queries.
Definition 2.2 (Inference Dimension) .
The inference dimension of ( X, H ) with query set Q is the smallest k such that for any subset S ⊂ X of size k , ∀ h ∈ H , ∃ x ∈ S s.t.
Q h ( S \ { x } ) infers x .
If no such k exists, then we say the inference dimension is ∞ .
KLMZ proved that query-efficient learning is possible if and only if inference dimension is finite.
Theorem 2.3 (Inference Dimension Characterizes Active Learning [3, Theorem 1.5]) .
Let ( X, H ) be a class with inference dimension k with respect to query set Q .
The expected query complexity of perfectly learning ( X, H ) is[-
-]
[Equation]
Similarly, the query complexity of active learning ( X, H ) is at most
[Equation]
and if k = ∞ , active learning gives no asymptotic improvement over standard passive bounds: [Equation] more natural model is Rivest and Sloan’s [16] Reliable and Probably Useful
(RPU) Learning, where the learner can abstain with low probability but may never err.
Perfect learning finite samples is essentially equivalent to the RPU model in most settings by standard generalization techniques, including all settings we study.
Worst-Case Active Learning PTFs 3.1 Upper Bounds
In this section, we sketch the proofs of our upper bounds Theorem 1.1 and Theorem 1.2.
At a high level, both results follow from the fact that a PTF f can be query-efficiently broken into a small number of monotone segments which act like thresholds.
This is done in two main steps.
First, we observe that it is possible to break f into a small number of segments sharing the same sign pattern.
Lemma 3.1.
For any degree- d polynomial f ∈ H d and set S = { s 1 ≤ . . .
≤ s n } , given sign ( f ( i ) ( x )) for all 1 ≤ i ≤ k and x ∈ S , it is possible to partition S into j ≤ O ( d 2 ) contiguous, disjoint segments
[Equation]
such that each interval has a fixed sign pattern, i.e. for every 1 ≤ ℓ ≤ j and s, s ′ ∈ I ℓ :
[Equation] s [Equation] f
[Equation] = SgnPat [Equation]
s [Equation] f (1) ) .
Moreover, this can be done in O ( n ( d + log n )) time.
Second, we observe that f must be monotone on any interval with a fixed pattern.
Lemma 3.2.
If f ∈ H d and a < b ∈ R satisfy SgnPat ( f (1) , a ) =
SgnPat ( f (1) , b ) , then f is monotone on [ a, b ] .
Based on these facts, it is not hard to see that Theorem 1.1 is realized by the following iterative algorithm that learns each derivative in a top-down fashion until reaching f (0) itself.
Result S
Apply Lemma 3.
The iterative approach gives a simple, deterministic technique for learning PTFs with derivative queries, but comes at the cost of a high level of adaptivity.
We now give a simple randomized algorithm that smoothly interpolates between the query-efficient and low-adaptivity regimes.
Algorithm 2 is a batch variant of KLMZ’s original algorithm underlying Theorem 2.3.
The following batch variant of their upper bound follows from similar analysis (setting m above to 2 kn α ).
Theorem 3.3 (Inference Dimension → Batch Active Learning) .
Let ( X, H ) be a class with inference dimension k with respect to query set Q .
Then for any n ∈ N and α ∈ (1 / log( n ) , 1] and size n -subset S ⊂ X , B ATCH -KLMZ labels all of S in
[Equation]
expected queries, and only
[Equation]
expected rounds of adaptivity, where Q tot ( m ) is the number of queries available on a set of m points.
To prove Theorem 1.2, it therefore suffices to show that PTFs have bounded inference dimension with respect to derivative queries.
In fact, this is essentially immediate from Lemma 3.1 and Lemma 3.2.
Any f ∈ H d can be broken up into O ( d 2 ) monotonic regions based on sign pattern.
Given any three points in such a region the middle can always be inferred, so by the pigeonhole principle we have:
Lemma 3.4.
The inference dimension of ( R , H d ) with derivative queries is O ( d 2 ) .
In the supplementary materials, we additionally prove an Ω( d ) lower bound on inference dimension. Algorithm Result:
Labels all points in S
Input: Class ( X, H ) , Subset S ⊆ X , Query set Q , Query Oracle O Q
Parameters: Inference dimension k , Batch size m , Iteration cutoff t = log( n ) log( m 2 k ) Algorithm: 8 S 0 ← S 9 for i in range t do T ← {} while Q h ( T ) infers less than a m − 2 k m fraction of S i do Sample T ∼ S mi
Query T : Q h ( T ) ← O Q ( T ) end
S i +1 ← { x ∈ S i : Q h ( T ) does not infer x } if | S
i +1 | ≤ m then
Query O Q ( S i +1 )
Return end 20 end
3.2 Lower Bounds
We briefly discuss the proof techniques behind our lower bounds in Theorem 1.1 and Theorem 1.3.
The former follows from a standard information theoretic argument, noting that degree- d PTFs result in n Ω( d ) possible labelings of any n point subset.
For the latter, let Q ˆ i denote the family of derivative queries without the i th derivative.
Theorem 2.3 shows it is enough to prove the inference dimension of ( R , H d ) with respect to Q ˆ i is infinite.
By a simple induction, it is sufficient to consider i =
d − 1 .
This is done by exhibiting for all n ∈ N a family of PTFs { h, p 1 , . . .
, p n } and points
S =
{ s 1 , . . .
, s n } such that h and p j are indistinguishable on S \ { s j } with respect to ˆ Q d − 1 , but h ( s j ) ̸
= p j
( s j ) .
We achieve such a construction by building a set of polynomials where each p ( d − 1) i is sufficiently negative around s i to force the function to flip sign, but the points are sufficiently spread out that this cannot be detected on any other s j for j ̸
= i .
Average-Case Active Learning PTFs
We now sketch the proof of our average-case results.
We briefly recall the model: given distributions
D X over R and D H over H d , we are interested in analyzing the expected number of queries needed to infer all labels of a sample S ∼ D X with respect to h ∼ D H .
We now present a simple generic algorithm for this problem we call “Sample and Search.”
Given a sample S ⊂ R :
Query the label (sign) of points from S uniformly at random until either: (a) We have queried all n points in S .
(b) We see d sign flips in the queried points,
i
e. we have queried x 1 , . . .
, x k and there exists indices i 1 < . . .
<
i
d +1 such that sign ( f ( x i j ))
̸ = sign ( f ( x i j +1 )) for all j = 1 , . . .
, d . 2.
If (b) occurred in the previous step, perform binary search on the points in S between each pair ( x i j , x i j +1 ) to find the sign threshold (and thereby labels) in that interval.
It is not hard to see Sample and Search infers all labels of points in S by construction.
The main challenge lies in analyzing its query complexity, and in particular step 1 which can be thought of as a variant of the classical coupon collector (CC) problem.
In our setting, the “coupons” are made up by the intervals between adjacent roots, and their probability is given by the mass of the marginal distribution on that interval.
With this in mind, let Y be the random variable measuring the number of samples required to hit each interval (coupon) at least once, and let Z = min( Y, n ) .
Proposition 4.1.
The expected query complexity of the Sample and Search Algorithm is at most:
[Equation]
It is worth noting that E D X ,D H [ Z ] and E D X ,
D H
[ Y ] can differ drastically.
As a basic example, consider the case where d = 1 and we draw our n points and one root uniformly at random from [0 , 1] .
It is a simple exercise to show that E D X ,D H
[ Y ] = ∞ , whereas E D X ,D H
[ Z ] =
O (log n ) .
With this in mind, we finish the section by sketching the proof of our average-case results.
We focus on the setting of the uniform distribution (Theorem 1.4).
The Dirichlet case (Theorem 1.5) follows similar overall ideas, but requires more involved calculation and machinery to deal with dependence of the roots.
In both cases, however, the first step is to observe the following standard bound on Y conditional on the roots being well separated.
Lemma 4.2.
For any x ∈ R + , let E x denote the event that f ∼ D H has measure at least 1 x over
D X between any two adjacent roots, the leftmost root and 0 , and the rightmost root and 1 .
Then:
[Equation]
To analyze the capped variable Z = min { Y, n } , we expand the expectation and cap the integral:
E D X ,D H
[ Z ] =
[Equation]
By Lemma 4.2, the righthand probability is lower bounded by the probability the minimum interval measure (denoted M) is Ω( log d x ) , which can be computed directly in the uniform setting:
[Equation]
Plugging this back into Equation (1) and computing the integral gives E [ Z ] ≤ O ( d 2 log( n )) .
To prove the corresponding lower bound for this problem (appearing in Theorem 1.4), we appeal to classic information theoretic techniques.
In particular, the expected number of binary queries to reveal the labels of S cannot be less than the entropy of the resulting distribution over labelings.
In the uniform case, one can directly show the entropy is Ω( d log( n )) , so Sample and Search is off from the information theoretic optimum by at most a factor of d .
We briefly remark on the challenges moving to the Dirichlet distribution.
Due to the dependence of roots in this setting, bounding the minimum measure M in Equation (2) becomes substantially more difficult.
For small α , we union bound over each variable and directly analyze the marginal distributions.
For large α , we appeal to strong Bernstein-type concentration bounds on Beta distributions of Skorski [49].
The lower bound follows from a similar appeal to information theoretic techniques.
The trick in this case is to observe the induced label distribution is exactly the well-studied “Dirichlet-Multinomial” distribution whose asymptotic entropy is known (see e.g. [50, Theorem 2]).
Beyond Univariate PTFs
Finally, we sketch the proof of Theorem 1.6, our lower bound on learning two-variate quadratics with derivative queries.
Recall in this setting the learner is allowed to make gradient queries of the form sign (cid:16) ∂f∂x ( x 1 , y 1 ) , ∂f∂y ( x 1 , y 1 ) (cid:17) , and Hessian queries of the form sign (cid:16) ∂ 2 f ∂x∂x ( x 1 , y 1 ) , ∂ 2 f ∂x∂y ( x 1 , y 1 ) , ∂ 2 f ∂y∂x ( x 1 , y 1 ) , ∂ 2 f ∂y∂y ( x 1 , y 1 ) (cid:17) for any ( x 1 , y 1 ) ∈ R 2 in the learner’s sample (along with standard label queries).
As in our worst-case lower bound for missing derivatives, we prove the inference dimension of this class is infinite.
Our construction consists of n points distributed evenly on the quarter circle of radius 1 in the first quadrant.
Consider the functions − x 2 − y 2 ± ϵxy (with ϵ small enough), which have negative values for all label queries, gradient queries, and diagonal entries of the Hessian, and either all positive or all negative values for the off-diagonal.
The goal is then to find functions h i which flip the label on the i th point, but are indistinguishable from one of − x 2 − y 2 ± ϵxy elsewhere (at least half the points are therefore indistinguishable from one of these choices, which gives an inference dimension lower bound of n/ 2 for any n ∈ N ).
The idea is to consider rotations of the function f = xy − c 1 y 2 , which is only positive for a small sector dependent on c 1 .
h i is chosen to be the rotation where this sector contains only the i th point.
To ensure the derivatives and diagonal Hessian values remain negative, we subtract c 2 ( x 2 + y 2 − 1) which does not change the value on the unit circle.
Since the Hessian is symmetric for these functions, they also agree with one of − x 2 − y 2 ±
ϵxy on the off-diagonal giving the desired result.