Active Online Domain Adaptation
Abstract
Online machine learning systems need to adapt to domain shifts.
Meanwhile, acquiring label at every timestep is expensive.
We propose a surprisingly simple algorithm that adaptively balances its regret and its number of label queries in settings where the data streams are from a mixture of hidden domains.
For online linear regression with oblivious adversaries, we provide a tight tradeoff that depends on the durations and dimensionalities of the hidden domains.
Our algorithm can adaptively deal with interleaving spans of inputs from different domains.
We also generalize our results to non-linear regression for hypothesis classes with bounded eluder dimension and adaptive adversaries.
Experiments on synthetic and realistic datasets demonstrate that our algorithm achieves lower regret than uniform queries and greedy queries with equal labeling budget.
1. Introduction
Domain shift, the difference between training and testing distributions, is a major bottleneck for many machine learning applications (Kouw & Loog, 2018).
For example, bioimaging model learned from one hospital may not transfer to machines in another hospital (Xu & Yang, 2011).
Even imperceptible natural distributional shift can cause big drops in accuracy for many existing models (Recht et al., 2018; 2019; Engstrom et al., 2020).
Principled algorithms for domain adaptation require strong assumptions on the types of domain shift (Ben-David et al., 2010).
On the other hand, learning the shift requires strong generative models (Hoffman et al., 2017).
Online learning is a classical theoretical framework to deal with worst-case domain shift (Hazan, 2019).
In online learning, even though the data is assumed to be given adversarially, strong regret bounds are attainable for many problems.
So far, practical deployments of fully online learning systems has been somewhat limited, because labels are expensive to obtain; see Strickland (2018) for an example in fake news detection.
Even if a system only acquires a small set of labels periodically, a label budget linear in time is still a luxury.
Cesa-Bianchi et al.
(2004b) study label-efﬁcient online learning for prediction with expert advice.
Their algorithm queries the label of every example with a ﬁxed probability, which, as they show, achieves minimax-optimal regret and query complexity for this problem.
However, querying with uniform probability does not take into account the algorithm’s uncertainty on each individual example, and thus can be suboptimal when the problem has certain favorable structures.
For example, a sequence of online news may come from the mixture of a few topics or trends, and some news topics may require more samples to categorize well compared to others.
We aim to improve label-efﬁciency in online regression by exploiting hidden domain structures in the data.
We assume that each input is from one of m potentially overlapping domains.
For each input, the learner makes a prediction, incurs a loss, and decides whether to query its label.
The regret of the learner is deﬁned as the difference between its cumulative loss and that of the best predictor in hindsight.
We assume realizability, i.e., there exists a predictor that is Bayes optimal across all the domains.
This is a reasonable assumption in modern machine learning, since features can be high-dimensional (so that different domain may rely on different features), and models are often overparameterized (Zhang et al., 2016).
Our goal is to trade off between regret and query complexity: given a ﬁxed label budget, we hope to achieve a regret as low as possible.
We propose QuFUR (Query in the Face of Uncertainty for
Regression), a surprisingly simple query scheme based on uncertainty quantiﬁcation.
We start with online linear regression from R d to R with an oblivious adversary.
In the realizable setting, with additional regularity conditions, we provide the following regret guarantee of QuFUR with label budget B : for any partition of [ T ] into domains, I 1 , . .
, I m , if for every u in [ m ] , the u -th domain S u
= { x t : t ∈ I u } has T u examples and lies in a d u -dimensional subspace of R d , the regret is ˜ O
(( (cid:80) mu =1 √
d u T u ) 2 /B )
(Theorem 2).
When choosing m = 1 and I 1 =
[ T ] , we see that the regret of QuFUR is at most ˜ O ( dT/B ) , matching minimax lower bounds (Theorem 7) in this setting.
The advantage of QuFUR’s adaptive regret guarantees becomes signiﬁcant when the domains have heterogeneous time spans and dimensions.
For example, ( (cid:80) mu =1 √ d u T u ) 2 can be substantially less than dT when the T u /d u ’s are heterogeneous across different u ’s. As an example, if d = m , d 1 = d 2 = . . .
=
d m = 1 , T 1 = T − m + 1 , T 2 = . . .
=
T m = 1 , ( (cid:80) mu =1 √ d u T u ) 2 =
O (cid:0) T + m 2 (cid:1) , which is substantially less than dT =
O ( mT ) when 1 (cid:28) m (cid:28) T .
Using standard online-to-batch conversion (Cesa-Bianchi et al., 2004a), we also obtain novel results in batch active learning for regression (Theorem 12).
Furthermore, we also deﬁne a stronger notion of minimax optimality, namely hidden domain minimax optimality , and show that QuFUR is optimal in this sense (Theorem 3), for a wide range of domain structure speciﬁcations.
We generalize our results to online regression with general hypothesis classes against an adaptive adversary.
We obtain a similar regret-query complexity tradeoff, where the analogue of d u is (roughly) the eluder dimension (Russo &
Van Roy, 2013) of the hypotheses class with respect to the support of domain u (Theorem 4).
Experimentally, we show that our algorithm outperforms the baselines of uniform and greedy query strategies, in a synthetic experiment and several LIBSVM datasets (Chang & Lin, 2011).
2. Related works
Active learning.
We refer the readers to Balcan et al.
(2009); Hanneke (2014); Dasgupta et al. (2008); Beygelzimer et al. (2010) and the references therein for background on active learning. For classiﬁcation, a line of work (Dasgupta & Hsu, 2008; Minsker, 2012; Kpotufe et al., 2015;
Locatelli et al., 2017) performs hierarchical sampling for nonparametric active learning.
The main idea is to maintain a hierarchical partitioning over the instance domain (either a pre-deﬁned dyadic partition or a pre-clustering over the data), and “zooms” into uncertainty regions whose conﬁdence interval of P ( y = 1 | x ) contains 1 / 2 .
For regression, many works (Fedorov & Hackl, 2012; Chaudhuri et al., 2015) study the utility of active learning for maximum likelihood estimation in the realizable setting.
Recent works also study active linear regression in nonrealizable (Drineas et al., 2006; Derezinski et al., 2018; Derezi´nski & Warmuth, 2018; Sabato & Munos, 2014) and heteroscedastic (Chaudhuri et al., 2017; Fontaine et al., 2019) settings.
These works do not consider domain structures except for Sabato & Munos (2014), who propose a domain-aware active sampling scheme.
Their algorithm needs to know the domain partition a priori, and its performance depends on the quality of the partition.
The empirical works of Rai et al. (2010); Saha et al. (2011); Xiao & Guo (2013) study stream-based active learning when inputs comes from pre-speciﬁed source and target domains. Our algorithm handles multiple domains, and does not require knowledge of which domain the inputs come from.
Active online learning.
Earlier works on selective sampling when iid data arrives in a stream and a label querying decision has to be made after seeing each example (Cohn et al., 1994; Dasgupta et al., 2008; Hanneke, 2011) implicitly provide online regret and label complexity guarantees.
Works on worst-cast analysis of selective sampling for linear classiﬁcation (Cesa-Bianchi et al., 2006) provide regret guarantees similar to that of popular online linear classiﬁcation algorithms such as Perceptron and Winnow, but their label complexity guarantees are runtime-dependent and therefore cannot be easily converted to a guarantee that only involves problem parameters deﬁned apriori.
Subsequent works (Cesa-Bianchi et al., 2009; Dekel et al., 2010;
Cavallanti et al., 2011; Agarwal, 2013) study the setting where there is a parametric model on P ( y | x, θ ) with unknown parameter θ , and the x ’s shown can be adversarial.
Under those assumptions, they obtain regret and query complexity guarantees dependent on the fraction of examples with low margins.
Yang (2011) gives a worst-case analysis of active online learning for classiﬁcation with drifting distributions, under the assumption that the Bayes optimal classiﬁer is in the hypothesis class given to the learner.
In contrast, our work gives adaptive regret guarantees in terms of the hidden domain structure in the data, and focuses on regression instead of classiﬁcation.
KWIK model.
In the KWIK model (Li et al., 2011), at each time step, the algorithm is asked to either query the label (and output “Don’t know”) or predict an output with at most (cid:15) error.
In contrast, in our setting, the learner’s goal is to minimize its cumulative regret, as opposed to making pointwise-accurate predictions.
Cesa-Bianchi et al.
(2009) study linear regression in the KWIK-model, and propose an algorithm similar to ours; unlike our work, they do not consider the algorithm’s adaptivity to domain structures.
Szita & Szepesv´ari (2011) propose an algorithm that works in an agnostic setting, where the error guarantee at every round depends on the agnosticity of the problem.
A relaxed
KWIK model that allows a prespeciﬁed number of mistakes has been studied in (Sayedi et al., 2010; Zhang & Chaudhuri, 2016).
Adaptive/Switching Regret. Adaptive regret (Hazan &
Seshadhri, 2007; Daniely et al., 2015) is the excessive loss of an online algorithm compared to the locally optimal solution over any continuous timespan.
Our algorithm can be interpreted as being competitive with the locally optimal solution on every domain, even if the timespans of the domains are not continuous, which is closer to the concept of switching regret with long-term memory studied in e.g, (Bousquet & Warmuth, 2002; Zheng et al., 2019).
However, typical bounds for this regret measure have a polynomial dependence on the number of domain switches, which does not appear in our bounds at all.
Online linear regression. Literature on fully-supervised online linear regression has a long history (Vovk, 2001;
Azoury & Warmuth, 2001).
As is implicit in Cesa-Bianchi et al. (2004b), we can reduce from fully-supervised online regression to active online regression by querying uniformly randomly with a ﬁxed probability.
Combining this reduction with existing online linear regression algorithms (Hazan et al., 2007), we get ˜ O ( dT/B ) regret with O ( B ) queries for any B ≤ T .
Our bound matches this in the realizable and oblivious setting when there is one domain, and is potentially better with more domain structures.
3. Setup and Preliminaries 3.1. Setup
Active online learning with squared loss Let F = { f :
X → [ − 1 , 1] } be a hypotheses class.
We consider the realizable setting where y t = f ∗ ( x t ) + ξ t for some f ∗ ∈
F and random noise ξ t .
The adversary decides f ∗ ∈ F before interaction starts.
ξ t ’s are independent zero-mean, sub-Gaussian random variables with variance proxy η 2 .
The learner is given a label budget B .
For each t = 1 , . . .
, T :
• x t is revealed to the learner.
• The learned predicts ˆ y t
= ˆ f t ( x t ) using predictor ˆ f t ∈ F , incurring loss (ˆ y t − y t ) 2 .
• The learner sets a query indicator q t ∈ { 0 , 1 } .
If q t = 1 , y t is revealed.
The performance of the learner is measured by its query complexity Q
= (cid:80) Tt =
1 q t and regret R =
(cid:80) Tt
=1 (ˆ y t − f ∗ ( x t )) 2 .
By our realizability assumption, our notion of regret coincides with the one usually used in online learning when taking expectations; see Appendix D. Our goal is to design a learner that has low regret R subject to its budget constraint Q ≤ B .
Oblivious vs. adaptive adversary
In the oblivious setting, the adversary decides the sequence { x t } Tt =1 beforehand.
In the adaptive setting, the adversary can choose x t depending on { x 1: t − 1 , f 1: t − 1 , ξ 1: t − 1 } .
Miscellaneous notations For a vector v ∈ R d and a positive semideﬁnite matrix M ∈ R d × d , (cid:107) v (cid:107) M := √ v (cid:62) Mv .
For vectors { z t } Tt =1 ⊆ R l , and S =
{ i 1 , . .
, i n } ⊆
[ T ] , denote by Z S the n × l matrix whose rows are z (cid:62)
i 1 , . .
, z (cid:62) i n .
Deﬁne clip( z ) :
= min(1 , max( − 1 , z )) and ˜ η := max { 1 , η } .
3.2. Baselines
We ﬁrst study linear regression with oblivious adversary, and generalize to non-linear case with adaptive adversary in Section 5.
For now, hypothesis class F = {(cid:104) x, θ (cid:105) : θ ∈ R d , (cid:107) θ (cid:107) 2 ≤ C } .
Let the ground truth hypothesis be θ ∗ , and input space X be a subset of { x ∈ R d : (cid:107) x (cid:107) 2 ≤ 1 , (cid:104) [Equation]
[Equation]
Uniform query is minimax-optimal with no domain structure As a baseline, consider the algorithm that always queries and returns the regularized least squared estimator [Equation] where λ = 1 /C 2 .
It is well-known (Vovk, 1990;
Azoury & Warmuth, 2001) that this fully-supervised algorithm achieves R = ˜
O
(˜ η 2 d ) with Q = T .
Consider an active learning extension of the above algorithm that queries uniformly randomly with probability µ , and always predicts with the regularized least squared estimator computed based on all queried examples ˆ θ t = argmin θ (cid:80)
i ∈
[ t − 1] ,q i =1 ( (cid:104) θ, x i (cid:105) − y i )
+ λ (cid:107) θ (cid:107) 2 .
We show that the above uniform querying strategy achieves R = ˜
O (˜ η 2 dT/B ) with Q = ˜
O ( B ) =
˜ O ( µT ) in Appendix A.4.
As shown in Theorem 7, this tradeoff is minimax optimal if ˜ η is a constant.
Although this guarantee is optimal in the worst case, one major weakness is that it is too pessimistic: as we will see next, when the data has some hidden domain structure, it is possible to achieve substantially better regret guarantees than the worst-case ones, if the learner has access to auxiliary domain information.
Oracle baseline when domain structure is known Suppose the learner is given the following knowledge from an oracle: there are m domains; for each u in [ m ] , there are a total of T u examples from domain u from a subspace of R d dimension d u .
In addition, for every t , the learner is given the index of the domain example x t comes from.
In this setting, the learner can use the following domain-aware querying scheme: for any example in domain u , it queries its label independently with probability µ u ∈ (0 , 1] .
Within domain u , the learner incurs O ( µ u T u ) queries and ˜ O
(˜ η 2 d u /µ u ) regret.
Summing over domains, its achieves a label complexity of
O ( (cid:80) mu =1 µ u T u ) , and a regret of ˜ O
(˜ η 2 (cid:80) mu =1 d u /µ u ) .
This motivates the following optimization problem: min µ
[Equation]
t. [Equation]
i.e., we choose domain-dependent query probabilities that minimize the learner’s regret guarantee, subject to its query complexity being at most B .
When B ≤ (cid:80) mu =1 √ d u T u · min u (cid:112) T u /d u , the optimal µ u is proportional to (cid:112) d u /T u .
This yields a regret guarantee of O (˜ η 2 ( (cid:80) u √ d u T u ) 2 /B ) .
Although this strategy can sometimes achieve much smaller regret than uniform query (as we have seen in Section 1, ( (cid:80) u √ d u T u ) 2 can be substantially smaller than dT ), it still has two crucial drawbacks: ﬁrst, it is not clear if this guarantee is always no worse than uniform querying, especially when (cid:80) m u =1 d u (cid:29) d ; second, the domain membership of each example is rarely known in practice.
We will develop algorithms that overcome these drawbacks.
4. Active online domain adaptation for linear regression: algorithms, analysis, and matching lower bounds
We start by presenting an algorithm parameterized by α in Section 4.1, which has a natural cost minimization interpretation.
[-Uncertainty estimate ∆ t ← ˜ η 2 min { 1 , (cid:107) x t (cid:107) 
-]
We then present a ﬁxed-budget variant of it in Section 4.2.
Section 4.3 shows that our algorithm is minimax-optimal under a wide range of domain structure speciﬁcations.
4.1. Main Algorithm: Query in the Face of Uncertainty for Regression (QuFUR)
We propose QuFUR (Query in the Face of Uncertainty for
Regression), namely Algorithm 1.
At each time step t , the algorithm ﬁrst computes ˆ θ t , a regularized empirical risk minimizer on the labeled data obtained so far, then predict using ˆ f t ( x ) =
clip( (cid:104)
ˆ θ t , x (cid:105) ) .
It makes label queries with probability proportional to a high-probability upper bound of the instantaneous regret (ˆ y t − (cid:104) θ ∗ , x t (cid:105) ) 2 , which can also be interpreted as the uncertainty on x t .
Intuitively, when the algorithm is already conﬁdent about the current prediction, it will save budget for learning from less certain inputs in the future.
QuFUR measures the uncertainty of x t using ∆ t :=
˜ η 2 min (cid:18) 1 , (cid:107) x t (cid:107) 2 M − 1 t (cid:19) , where M t = λI + (cid:80) i ∈Q t x i x (cid:62) i , and Q t is the set of labeled examples seen up to time step t − 1 .
We will show in Lemma 1 that with high probability, the instantaneous regret on x t is at most ˜ O (∆ t ) .
QuFUR queries the label y t with probability min { 1 , α ∆ t } , where α is a parameter that tradeoffs query complexity and regret.
Require: Total dimension d , time horizon T , θ ∗ ’s norm bound C , noise level η , parameter α .
1: M ← 1 C 2 I , queried dataset Q ← ∅ . 2: for t = 1 to T do 3: Compute regularized least squares solution ˆ θ t ← M − 1 X (cid:62)Q Y Q .
Let ˆ f t ( x ) =
clip( (cid:104)
ˆ θ t , x (cid:105) ) be the predictor at time t , and predict ˆ y t ← ˆ f t ( x t ) .
{+Uncertainty estimate ∆ t ← ˜ η 2 min { 1 , (cid:107) x t (cid:107) 
+}
M − 1 } .
With probability min { 1 , α ∆ t } , set q t ← 1 ; otherwise q t ← 0 .
7: if q t = 1 then 8:
Query y t .
M ← M + x t x (cid:62) t , Q ← Q (cid:83) { t } .
Perhaps surprisingly, the simple query strategy of QuFUR can leverage hidden domain structure, as shown by the following theorem.
Theorem 1.
Suppose the example sequence { x t } Tt =1 has the following structure: [ T ] can be partitioned into m disjoint nonempty subsets { I u } mu =1 , where for each u , | I u | = T u , and { x t } t ∈ I u lie in a subspace of dimension d u . Suppose α ∈
[Equation]
If Algorithm 1 receives inputs dimension d , time horizon
T , norm bound C , noise level η , parameter α , then, with probability 1 − δ :
Its query complexity Q = ˜
O (˜ η · √ α (cid:80) u √ d u T u ) .
Its regret R = ˜
O
(˜ η · (cid:80) u √ d u T u / √ α ) .
The proof of the theorem is deferred to Section A.1.
We make a series of remarks below:
Novel notion of adaptive regret. The above tradeoff is novel in that it holds for any meaningful domain partition.
Our proof actually shows that for any (not necessarily contiguous) subsequence I ⊆
[ T ] , QuFUR has Q = ˜
O (˜ η · (cid:112) d I | I |·√ α ) and R =
˜ O
(˜ η (cid:112) d I | I | ) / √ α within
I , where d I is the dimension of span( { x t : t ∈ I } ) .
This type of guarantee is stronger than the adaptive regret guarantees provided by e.g. (Hazan & Seshadhri, 2007), where the regret guarantee are only with respect to continuous intervals.
Matching uniform querying baseline and minimax optimality. Our tradeoff is never worse than the uniform querying baseline; this can be seen by applying the theorem with the trivial partition { [ T ] } yields Q = ˜
O
(˜ η √ αdT ) and
R = ˜
O
(˜ η (cid:112) dT/α ) .
Therefore, same as the uniform query baseline, this guarantee is also minimax optimal for constant η , in light of Theorem 7 in Appendix B.2.
Matching oracle baseline.
QuFUR matches the domainaware oracle baseline even without prior knowledge of domain structure.
We show in Theorem 3 that in a wide range of problem speciﬁcations, this baseline, as well as QuFUR, is minimax-optimal in our problem formulation with domain structure.
Fixed-cost-ratio interpretation.
The tradeoff in Theorem 1 can be interpreted in a ﬁxed-cost-ratio formulation.
Suppose a practitioner decides that the cost ratio between 1 unit of loss and 1 label query is c : 1 .
The performance of the algorithm is then measured by its total cost cR + Q .
Theorem 1 shows that QuFUR( α ) balances the cost incurred by prediction and the cost incurred by label queries, in that Q ≈ αR .
We show in Appendix C that QuFUR with input
α = c achieves near-optimal total cost, for a wide range of domain structure parameters.
Dependence on η .
Our query complexity and regret bounds have a dependence on ˜ η = max( η, 1) .
Similar dependence also appears in online least-squares regression literature (Vovk, 1990; Azoury & Warmuth, 2001). Requirements on α .
The requirement that α ≥ 1 / ˜ η 2 (cid:16) 1 / ( (cid:80) u √ d u T u ) (cid:17) 2 is immaterial: if α is smaller than this threshold, the regret guarantee is at least ˜
O (˜ η 2 ( (cid:80) u √ d u T u ) 2 ) ; as ˜ η 2 ( (cid:80) u √ d u T u ) 2 ≥ T , the above guarantee can in fact be trivially achieved by an algorithm that performs no label queries and always predicts ˆ y t = 0 . The requirement that α ≤ 1 / ˜ η 2 min u ∈ [ m ] T u /d u corresponds to a label usage of order ˜ O (cid:16)(cid:80)
u √ d u T u · min u ∈ [ m ] (cid:112) T u /d u (cid:17) , which also matches the label budget range of the oracle baseline discussed in
Section 3.2.
Running time. The most computationally intensive operation in QuFUR is the d × d matrix inversion in line 8, which occurs T times.
We can apply the same optimization as in Abbasi-Yadkori et al.
(2011), i.e. recompute ˆ θ t only when det( M t ) increases by a constant factor.
Using this, only O ( d log T ) matrix inversions are required; the regret of the modiﬁed algorithm is of the same order as the original algorithm, up to log factors.
4.2. QuFUR with a ﬁxed label budget
The label complexity bound in Theorem 1 involve parameters { d u , T u } mu =1 , which may be unknown in advance.
In many practical settings, the learner is given a label budget
B . For such settings, we propose a ﬁxed-budget version of QuFUR, Algorithm 2, that takes B as input, and achieves near-optimal regret bound subject to the budget constraint, under a wide range of domain structure speciﬁcations. Algorithm
Require: Total dimension d , time horizon T , label budget
B , θ ∗ ’s norm bound C , noise level η .
1: Number of copies k ← (cid:100) 4 log 2 T (cid:101) .
2: for i = 0 to k do 3:
Parameter α
i ← 2 i /T .
4: Initialize M ← 1 C 2 I , Q ← ∅ .
5: for t = 1 to T do 6:
Compute regularized least squares solution ˆ θ t ←
M − 1 X (cid:62)Q Y Q .
Let ˆ f t ( x ) =
clip( (cid:104)
ˆ θ t , x (cid:105) ) be the predictor at time t , and predict ˆ y t ← ˆ f t ( x t ) .
Uncertainty estimate ∆ t ← ˜ η 2 min { 1 , (cid:107) x t (cid:107)
M − 1 } .
9: for i = 0 to k do 10: if (cid:80) t − 1 j =1 q ij < (cid:98) B/k (cid:99) then 11:
With probability min { 1 , α i ∆ t } , set q it = 1 .
12: if (cid:80) i q i t > 0 then 13:
Query y t .
M ← M + x t x (cid:62) t , Q ← Q (cid:83) { t } .
Algorithm 2 is a master algorithm that aggregates over k =
O (log T ) copies of QuFUR ( α ).
Each copy uses a different value of α lying in an exponentially increasing grid { 2 i /T 3 : i = 0 , . .
. , k } .
The grid ensures that each copy still has label budget (cid:98) B/k (cid:99) = ˜Ω( B ) , and there is always a copy that takes full advantage of its budget to achieve low regret.
The algorithm queries whenever one of the copies issues a query, and predicts using a model learned on all historical labeled data.
A copy can no longer query when its budget is exhausted.
The regret of the master algorithm is no worse that of the copy running on a parameter α i that make ˜Θ( B ) queries; this insight yields the following theorem.
Theorem 2.
Suppose the example sequence { x t } Tt =1 has the following structure: [ T ] can be partitioned into m disjoint nonempty subsets { I u } mu =1 , where for each u , | I u | = T u , and { x t } t ∈ I u lie in a subspace of dimension d u .
Moreover, integer B satisﬁes
[Equation]
If Algorithm 2 receives inputs dimension d , time horizon T , label budget B , norm bound C , noise level η , then:
Its query complexity Q is at most B .
With probability
[Equation]
The proof of the theorem is deferred to Appendix A.2.
We now compare the guarantee of QuFUR with the that of of the oracle baseline in Section 3.2: for any budget B ∈ [0 , ˜ O
( (cid:80) u √ d u T u min u (cid:112) T u /d u )] , Fixed-Budget QuFUR achieves a regret guarantee no worse than that of domain-aware uniform sampling, while being agnostic to { d u , T u } mu =1 and the domain memberships of the examples.
4.3. Lower bound
Our development so far establishes domain structureaware regret upper bounds
[Equation]
, achieved by Fixed-Budget QuFUR and domain-aware uniform sampling baseline (the latter requires extra knowledge about the domain structure and domain membership of each example, whereas the former does not).
In this section, we study optimality properties of the above upper bounds.
Speciﬁcally, we show via Theorem 3 that they are tight up to logarithmic factors, for a wide range of domain structure speciﬁcations.
Its proof can be found in Appendix B.1.
Theorem 3.
For any noise level η ≥ 1 , set of positive integers (cid:8) ( d u , T u ) (cid:9) m u =1 and integer B that satisfy
[Equation]
there exists an oblivious adversary such that:
It uses a ground truth linear predictor θ (cid:63) ∈ R
d such that (cid:107) θ ∗ (cid:107) 2 ≤ √ d , and for all t , (cid:12)(cid:12) (cid:104) θ ∗ , x t (cid:105) (cid:12)(cid:12) ≤ 1 ; in addition, the noises { ξ t } Tt =1 are sub-Gaussian with variance proxy η 2 .
It shows example sequence { x t } Tt =1 such that [ T ] can be partitioned into m disjoint nonempty subsets { I u } mu =1 , where for each u , | I u | = T u , and { x t } t ∈ I u lie in a subspace of dimension d u . 3.
Any online active learning algorithm A with label budget B has regret Ω(( (cid:80) mu =1 √ d u
[Equation]
The above theorem is a domain structure-aware reﬁnement of the Ω( dT/B ) minimax lower bound (Theorem 7 in Appendix B.2), in that it further constrains the adversary to present sequences of examples with domain structure parametrized by { d u , T u } mu =1 .
In fact, Ω( dT/B ) minimax lower bound is a special case of the lower bound of Theorem 3, by taking m = 1 , d 1 = d , and T 1 = T .
To discuss the tightness of the upper and lower bounds we obtained so far in more detail, we ﬁrst set up some useful notations.
Denote by E( { T u , d u } mu =1 ) the set of oblivious adversaries that shows example sequences { x t } Tt =1 , such that [ T ] can be decomposed to m subsets { I u } mu =1 , where for every u in [ m ] , | I u | = T u , and examples in subset { x t : t ∈ I u } lie in a subspace of dimension d u .
Additionally, denote by A( B ) the set of online active learning algorithms that uses a label budget of B .
Finally, for an algorithm A and an oblivious adversary E , deﬁne R( A , E ) as the expected regret of A in the environment induced by E . Consider the regime when the noise variance proxy η ∈ [1 , O (1)] . In this case, Theorem 3 shows that for all [Equation]
u } mu [Equation] 1 and B such that 3 holds, we have [Equation]

[Equation]
On the other hand, Theorem 2 shows that for all { d u , T u } mu =1 and B such that (1) holds, we have
[Equation]
Fixed-Budget QuFUR ( B ) , E (cid:1) ≤ [Equation] O    
[Equation]
(cid:88) [Equation] (cid:112) [Equation]
/B   . This shows that, if η ∈ [1 , O (1)] , for a wide range of domain structure speciﬁcations { d u , T u } mu =1 and budgets B (i.e., B/ ( (cid:80) u √ d u T u ) ∈
[ (cid:112) max u d u /T u , (cid:112) min u T u /d u ] ), the regret guarantee of Fixed-Budget QuFUR is optimal; furthermore, the algorithm requires no knowlege on the domain structure.
We call this property of Fixed-Budget QuFUR its hidden-domain minimax optimality .
5. Extension to realizable non-linear regression with adaptive adversary
QuFUR’s design principle, querying with probability proportional to uncertainty estimates of unlabeled data, can be easily generalized to deal with other active online learning problems.
In summary, QuFUR uses a suitable loss upper bound to yield query-regret tradeoff adaptive to the complexities and durations of individual domains.
We now generalize QuFUR to non-linear regression with adaptive adversaries, extending Russo & Van Roy (2013).
In this section, we require the domain partition { I u : u ∈ [ m ] } to have a property we call admissibility .
Deﬁnition 1.
The partition (cid:8) I u : u ∈ [ m ] (cid:9) is admissible, if the domain membership of the t -th example, u t
= { u : t ∈ I u } depends on the interaction history up to t −and unlabeled example x t ; formally, u t is σ ( H t − 1 , x t ) - measurable.
Domain complexity measure. Analogous to the dimension of the support in linear regression, we use d (cid:48)
u =
dim Eu ( F , 1 /T 2 u ) , the eluder dimension of F with respect to domain u ∈ [ m ] with support X u ⊆ X .
Formally,
Deﬁnition 2.
An input x ∈ X is (cid:15) -dependent of on another set of inputs { x i } ni =1 ⊆ X with respect to F if for all f 1 , f 2 ∈ F , (cid:112)(cid:80) ni =1 ( f 1 ( x i ) − f 2 ( x i )) 2 ≤ (cid:15) = ⇒ f
1 ( x ) − f 2 ( x ) ≤ (cid:15) .
Require:
Hypothesis set F , time horizon T , parameters α, δ, η .
1: Labeled dataset Q ← ∅ .
2: for t = 1 to T do 3:
Predict ˆ f t ← argmin f ∈F (cid:80) i ∈Q ( f ( x i ) − y i ) 2 .
Conﬁdence set F t ← { f ∈ F : (cid:80) i ∈Q ( f ( x i ) − ˆ f t ( x i ))
2 ≤ β |Q| ( F , δ ) } , 5: where β k := 8 η 2 log (4 N ( F , 1 /T 2 , (cid:107) · (cid:107) ∞ ) /δ )
+ 2 k/T 2 (16 + (cid:112)
η 2 ln
(16 k 2 /δ )) .
Uncertainty ∆ t = sup f 1 ,f 2 ∈F t (cid:12)(cid:12) f 1 ( x t )
− f 2 ( x t ) (cid:12)(cid:12) 2 .
With probability min { 1 , α ∆ t } , set q t = 1 .
8: if q t = 1 then 9:
Query y t .
Q ← Q (cid:83) { t } .
Deﬁnition 3.
The (cid:15) -eluder dimension of F with respect to support X u , dim Eu ( F , (cid:15) ) , is deﬁned as the length of the longest sequence of elements in X u such that for some (cid:15) (cid:48) > (cid:15) , every element is (cid:15) (cid:48) -independent of of its predecessors.
The above domain-dependent eluder dimension notion captures how effective the potential value of acquiring a new label can be estimated from labeled examples in domain u .
The Algorithm. The master algorithm, Algorithm 4 in Appendix A.3, runs O (log T ) copies of Algorithm 3.
At round t , Algorithm 3 predicts using the empirical risk minimizer ˆ f t based on all previously queried examples. Same as Algorithm 1, Algorithm 3 queries with probability min { 1 , α ∆ t } , where ∆ t is an uncertainty measure of the algorithm on example x t .
To compute the uncertainty measure, it constructs a conﬁdence set F t , so that with high probability, the ground truth f ∗ ∈ F t for all t .
The uncertainty measure ∆ t is the squared maximum disagreement between two hypotheses in F t on x t .
It can be seen that with high probability, its regret and query complexity are bounded by O ( (cid:80)
Tt =1 ∆ t ) and O ( (cid:80) Tt =1 min { 1 , α ∆ t } ) , respectively.
We can bound the regret of the algorithm on example from domain u , in terms of domain complexity measure R u = ˜
O (˜ η 2 d (cid:48)
u log N ( F , T − 2 , (cid:107) · (cid:107) ∞ )) . Here N ( F , (cid:15), (cid:107) · (cid:107) ∞ ) is the (cid:15) - covering number of F with respect to (cid:107) · (cid:107) ∞ .
Speciﬁcally, we have the following theorem.
Theorem 4.
Suppose the example sequence { x t } Tt =1 has the following structure: [ T ] has an admissible partition (cid:8) I u : u ∈ [ m ] (cid:9) , where for each u , | I u | = T u , and the eluder dimension of F w.r.t. { x t } t ∈ I u is d (cid:48) u .
Then, given label budget B ≤ ˜ O ( (cid:80) u √ R u T u min u (cid:112) R u /T u ) , Algorithm 4 satisﬁes: It has query complexity Q ≤ B ; 2. With probability 1 − δ , its regret R = ˜
O (( (cid:80) u √ R u T u ) 2 / B ) .
The proof of the theorem can be found in Section A.3.
Specializing the theorem to linear hypothesis class F = {(cid:104) x, θ (cid:105) : θ ∈ R d , (cid:107) θ (cid:107) 2 ≤ 1 } , if X u is a subset of a d udimensional subspace of R d , we have dim Eu ( F , 1 /T 2 u ) =
˜
O ( d u ) , log N ( F , 1 /T 2 u , (cid:107) · (cid:107) ∞ ) =
˜ O ( d ) , implying R u = ˜
O
(˜ η 2 d u d ) , which in turn implies that R = ˜
O (˜ η 2 d ( (cid:80) u √ d u T u ) 2 /B ) .
Compared to Theorem 2, the additional factor d is due to increased difﬁculty with adaptive adversary.
6. Experiments
We test the query-regret tradeoffs of QuFUR, the uniform query baseline (Section 3.2), and naive greedy query (i.e., always querying until labeling budget is exhausted) on several linear regression tasks.
Datasets We create a synthetic dataset with 20 domains.
Each domain has either T u = 100 and d u = 6 , or T u
=and d u = 3 .
Inputs from each domain spans a random subset of d u out of d = 40 dimensions, with potential overlap between domains.
θ ∗ is a random vector on the unit sphere in R d , as are x i ’s from domain u in R d u .
Noise ξ t ’s are iid zero-mean Gaussian with variance
η 2 = 0 . 1 .
We also experiment on two real-world LIBSVM datasets (Chang & Lin, 2011) cpu-small and Abalone .
cpusmall uses 12 features, such as system reads/writes per second, to predict portion of time that cpu runs in user mode.
Abalone uses 8 features (physical measurements) to predict animal ages.
Results We run QuFUR( α ) for α ∈ [1 / 400 , 400] and uniform queries with probability µ ∈ [0 . 01 , 1] .
Figure 1 shows that QuFUR achieves the lowest total regret under the same labeling budget across all 3 datasets.
Notably, QuFUR’s advantage is more signiﬁcant on cpu-small .
We conjecture that this task has underlying domain structure, as different CPU usage modes may be predicted from a subset of metrics.
QuFUR potentially exploits this latent structure without knowledge of its existence.
7. Conclusion
We formulate a novel task of active online learning with latent domain structure.
We propose a surprisingly simple algorithm that adapts to domain shifts, and give matching upper and lower bounds in a wide range of domain structure speciﬁcations for linear regression.
The strategy is readily generalizable to other problems, as we did for non-linear regression, simply relying on a suitable uncertainty estimate for unlabeled data.
We believe that our problem and solution can spur future work on making active online learning more practical.
A. Proofs for upper bounds A.1. Proof of Theorem
We provide the proof of Theorem 1 in this section.
We focus on regret and query complexity bounds on one domain I u , and sum over domain u to obtain Theorem 1.
Recall that we deﬁne the interaction history between the learner and the environment up to time t be H t = { x 1: t , f 1: t , ξ 1: t } ; we abbreviate E [ ·| x t , H t − 1 ] as E t − 1 [ · ] .
The following lemma upper bounds the regret with sum of uncertainty estimates, ∆ t = ˜
η 2 min (cid:18) 1 , (cid:107) x t (cid:107) 2 M − 1 t (cid:19) .
A similar lemma has appeared in Cesa-Bianchi et al.
(2009, Lemma 1).
Lemma 1.
In the setting of Theorem 1, with probability 1 − δ 2 , for all t ∈
[ T ] , (ˆ y t − (cid:104) θ ∗ , x t (cid:105) )
= ˜ O (∆ t ) .
Proof of Lemma 1.
Denote the value of M, Q at the beginning of round t as M t , Q t .
Let λ = 1 /C 2 , V t =
M t − λI
=
(cid:80) s ∈Q t x s x (cid:62) s .
Therefore, ˆ θ t =
M − 1 t ( (cid:80) s ∈Q t x s y s )
=
M − 1 t ( V t θ ∗ + (cid:80) s ∈Q t ξ s x s ) , and (cid:104) x
t , ˆ [Equation] (cid:105) =
(cid:88) s
[Equation]
t
[Equation] The ﬁrst term is a sum over a set of independent sub-Gaussian random variables, so it is ( ησ ) 2 -sub-Gaussian with σ 2 = (cid:80) s ∈Q t x (cid:62) t M − 1 t x s x [Equation] s M − 1 t x t ≤ x (cid:62) t M − 1 t x t .
Deﬁne event
[Equation]
  .
By standard concentration of subgaussian random variables, we have P ( E t ) ≥ 1 − δ 2 T .
Deﬁne E =
∩
Tt =1 E t .
By union bound, we have P ( E ) ≥ 1 − δ 2 .
We henceforth condition on E happening, in which case the ﬁrst term of Equation (4) is bounded by η (cid:112) 2 ln (4 T/δ ) (cid:107) x t (cid:107) M − 1 t at every time step t .
Meanwhile, the second term of Equation (4) can be bounded by Cauchy-Schwarz:
[Equation]
which is at most (cid:107) x t (cid:107) M − 1 t , since (cid:107) θ ∗ (cid:107) 2 ≤ C and λ = 1 /C 2 .
Using the basic fact that ( A + B ) 2 ≤ 2 A 2 + 2 B 2 , ( (cid:104) x t , ˆ θ t (cid:105) − (cid:104) x t , θ ∗ (cid:105) ) 2 ≤ (4 η 2 ln (4 T/δ ) +
[Equation]
(cid:107) x t (cid:107)
[Equation]
− 1 t . Since ˆ y t = clip( (cid:104) x t , ˆ θ t (cid:105) ) ∈ [ − 1 , 1] and (cid:12)(cid:12) (cid:104) x
t , θ ∗ (cid:105) (cid:12)(cid:12) ≤ 1 , we also trivially have (ˆ y t − (cid:104) θ ∗ , x t (cid:105) ) 2 ≤ 4 .
Therefore,
[Equation]
The following lemma bounds the sum of uncertainty estimates for k queried examples in a domain:
Lemma 2.
Let a 1 , . .
, a k be k vectors in R d .
For i ∈ [ k ] , deﬁne N i = λI
+ (cid:80) i − 1 j =1 a j a (cid:62) j .
Then, for any S ⊆
[ k ] ,
[Equation]
2 N − 1 i (cid:19) ≤ ln(det( λI + (cid:80) i ∈ S a i a (cid:62) i ) / det( λI )) .
Proof of Lemma 2.
We denote by N i,S =
λI + (cid:80) j ∈ S : j ≤ i − 1 a j a (cid:62) j .
As S is a subset of [ k ] , we have that N i,S (cid:22) N i .
Consequently, (cid:107) a i (cid:107) N − 1 i ≤ (cid:107) a i (cid:107) N − 1 i,S . Therefore, (cid:88) [Equation] min
[Equation]
min
[Equation]
det( λI ) (cid:33) , where the second inequality is well-known (see e.g. Lattimore & Szepesv´ari, 2018, Lemma 19.4).
Proof of Theorem 1.
Let p t = min(1 , α ∆ t ) be the learner’s query probability at time t ; it is easy to see that E t − 1 [ q t ] =
p t .
Let random variable Z t = q t ∆ t .
We have the following simple facts:
[Equation]
For every u ∈ [ m ] , deﬁne event
[Equation] 
  [Equation] [Equation] Applying Freedman’s inequality to { Z t } t ∈ I u
(see e.g. Bartlett et al., 2008, Lemma 2), we have that P ( F u ) ≥ 1 − δ 4 m .
Similarly, deﬁne
G = 
 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
T (cid:88) t =
[Equation]
(cid:88)
[Equation]
= q t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ O   (cid:118)(cid:117)(cid:117)(cid:116) T (cid:88)
[Equation]
Applying Freedman’s inequality to { q t } t ∈ I u , we have that P ( G ) ≥ 1 − δ 4 .
Furthermore, deﬁne H = E ∩[-mu =
-] ( ∩ mu =1 F u ) ∩ G , where E is the event deﬁned in the proof of Lemma 1.
By union bound, P ( H ) ≥ 1 − δ .
We henceforth condition on H happening.
By the deﬁnition of F u , Solving for (cid:80) t ∈ I u p t ∆ t in Equation (5), we get that
[Equation]
Using Lemma 2 with { a i } ki =1 = { x t } t ∈Q T , and S
= I u
∩ Q T , we get that
[Equation]
In combination with Equation (7), we have (cid:80) t ∈
I
u p t ∆ t =
˜ O
(˜ η 2 d u ) .
We divide the examples in domain u into high and low risk subsets with index sets I u, + and I u, − (abbrev.
I +
and I − hereafter).
Formally,
[Equation]
We consider bounding the regrets and the query complexities in these two sets respectively:
For every t in I + , as p t = 1 , label y t is queried, so [Equation] ∈ I +
[Equation]
t ∈ I + q t ∆ t ≤ (cid:88) t ∈
I u q t ∆ t = ˜ O (˜ η 2 d u ) . Since for every t in I − , ∆ t > 1 /α , we have (cid:80) t ∈ I + ∆ t > | I + | /α .
This implies that (cid:80) t ∈ I + p t
=
|
I + | =
˜ O
( α ˜ η 2 d u ) .
For every t in I − , p t = α ∆ t .
Therefore, (cid:80) t ∈ I − α ∆ 2 t = (cid:80) [Equation] I [Equation]
p
[Equation] ∆ [Equation] ≤ (cid:80)
[Equation]
p [Equation] ∆ t = ˜ O
(˜ η 2 d u ) .
By CauchySchwarz, and the fact that | I − | ≤ T u , we get that (cid:80) t ∈ I − ∆ t ≤ (cid:113) | I − | · ( (cid:80) t ∈ I
[Equation]
[Equation] ) = [Equation] O (˜
[Equation]
(cid:112) d u T u /
[Equation]
Consequently, (cid:80) t [Equation] I
[Equation]
p
[Equation]
= (cid:80)
[Equation]
I [Equation] Summing over the two cases, we have
[Equation]
By the assumption that α ≤ 1˜ η 2 min u T u d u , for every u , we have α ˜ η 2 d u ≤ ˜ η √ αd u T u .
Therefore, the above bounds can be simpliﬁed to
[Equation]
For the query complexity, from the deﬁnition of event G , applying AM-GM inequality on Equation (6), we also have
[Equation]
where in the last equality we use the assumption that α ≥ 1˜ η 2 ·
[Equation]
For the regret guarantee, we have by the deﬁnition of event E and Lemma 1 that
[Equation]
Using the second inequality of Equation (8), we get
[Equation]
The theorem follows.
A.2. Proof of Theorem
Before going into the proof, we set up some useful notations.
Deﬁne I = { 0 , 1 , . . .
, k } as the index set of the α i ’s of interest.
Recall the number of copies k = 1 + (cid:100) 4 log T (cid:101) ≤ 2 + 4 log T .
Recall also that B (cid:48) =
(cid:98) B/k (cid:99) is the label budget for each copy.
Let p it = min(1 , α i ∆ t ) be the intended query probability of copy i at time step t ; let r it ∼ Bernoulli ( p it ) be the attempted query decision of copy i at time step t ; let A it = 1 (cid:104)(cid:80) t − 1 j =1 r ij
< B (cid:48) (cid:105) , i.e. the indicator that copy i has not reached its budget limit at time step t . Using this notation, the actual query decision of copy i , q it , can be written as r it A it .
We have the following useful observation that gives a sufﬁcient condition for copy i to be within its label budget:
Lemma 3.
Given i ∈ [ k ] , if (cid:80) Tt =
1 A it r it < B (cid:48) , the following items hold:
(cid:80) Tt =
1 r it < B (cid:48) .
For all t ∈ [ T ] , A it = 1 , i.e. copy i does not run of label budget throughout.
Proof.
Suppose for the sake of contradiction that (cid:80)
Tt =1 r it ≥ B (cid:48) .
Consider the ﬁrst B (cid:48) occurrences of r ij = 1 ; call them
J = { j 1 , . . .
, j B (cid:48) } .
It can be seen that for all j ∈ J , A ij = 1 .
Therefore,
[Equation]
which contradicts with the premise that (cid:80) Tt =
1 A it r it < B (cid:48) .
The second item immediately follows from the ﬁrst item, as (cid:80) Tj =1 r ij < B (cid:48) implies that (cid:80) t − 1 j =1 r ij < B (cid:48) for every t ∈ [ T ] .
Complementary to the above lemma, we can also see that for every i ∈ [ k ] , (cid:80) Tt =
1 A it r it =
(cid:80) Tt =
1 q it ≤ B (cid:48) is trivially true.
We next give a key lemma that generalizes Theorem 1, and upper bounds (cid:80) Tt =
1 A it r it for all i ’s beyond the above trivial B (cid:48) bound.
Lemma 4.
There exists C = polylog( T, 1 δ ) ≥ 1 , such that with probability 1 − δ/ 2 ,
T (cid:88)
[Equation]
=
[Equation]
∆
[Equation]
≤ C · ˜ η (cid:88)
[Equation]
d u T u / √ α i , and
T
[Equation]
for every i ∈ I such that α i ∈
[Equation]
(cid:35) .
Proof.
Applying Freedman’s inequality to the martingale difference sequence { A it ( r it − p it ) }
Tt =1 , we get that with probability 1 − δ/ 4 ,
T
[Equation]
Applying Freedman’s inequality to { A it ( r it − p it )
∆ t 1 [ t ∈ I u ] } Tt =1 , and take a union bound over all u ∈ [ m ] , we get that with probability 1 − δ/ 4 , (cid:88)
[Equation]
I u
[Equation]
Using Lemma 2 with { a i } ki =1 = { x t } t ∈Q T , and S
= I u
∩ Q T we get that, deterministically, (cid:80) t ∈ I u
A it r it ∆ t ≤ (cid:80) t ∈ I u q t ∆ t =
˜ O
(˜ η 2 d u ) .
So with probability 1 − δ/ 4 , (cid:88)
[Equation]
We henceforth condition on Equations (9) and (10) occuring, which happens with probability 1 − δ/ 2 by union bound.
Let I + = { t ∈ I u : α i ∆ j > 1 } , and I − =
I u − I
+ .
For I + , by Equation (10), (cid:80) t ∈ I
+ A ij ∆ j = ˜ O
(˜ η 2 d u ) =
⇒ (cid:80) j ∈
I
+ A ij p ij =
˜ O
( α i ˜
η
2 d u ) .
For I − , by Equation (10), (cid:80) j ∈ I − A ij α i ∆ 2 j = (cid:80) j ∈ I − A ij p
j ∆ j = ˜ O
(˜ η d u )
; this implies that (cid:80) j ∈
I − A ij ∆ j = ˜
O
(˜ η (cid:112) d u T u /α i ) .
In this event, we also have (cid:80) j ∈
I − A ij p ij = (cid:80)
j ∈
I − A ij α i ∆ j = ˜
O
(˜ η √ d u T u α i ) .
Summing over the two cases, we have
[Equation]
By the assumption that α i ≤ 1˜ η 2 min u T u d u , for every u , we have, α i ˜ η 2 d u ≤ ˜ η √ α i d u T u .
This implies that [Equation]
Summing over u ∈ [ m ] , we get
[Equation]
Therefore, using Equation (9), we have
[Equation]
where the last inequality uses the assumption that α i ≥ 1 ˜ η [Equation]
The lemma follows.
We are now ready to prove Theorem 2.
Proof of Theorem 2.
First, the query complexity of Fixed-Budget QuFUR is B by construction, as the algorithm maintains k copies of QuFUR, and each copy consumes at most B (cid:48) = (cid:98) B/k (cid:99) labels.
We now bound the regret of Fixed-Budget QuFUR.
We consider B = Ck ( (cid:80) u √ d u T u ) · min u ∈ [ m ] (cid:112) T u /d u = ˜ O (cid:16)(cid:80) u √ d u T u ) · min u ∈ [ m ] (cid:112) T u /d u (cid:17) , where C = polylog( T, 1 δ ) ≥ 1 is deﬁned in Lemma 4. We will show that if
B ∈ (0 , B ] , with probability 1 − δ , the regret of Fixed-Budget QuFUR is at most ˜ O
[Equation]
If B < 2 C ˜ η 2 , the regret of the algorithm is trivially upper bounded by 4 T , which is clearly ˜ O (cid:18) ˜
η 2 ( (cid:80) u √ d u T u ) 2 B (cid:19) .
Therefore, throughout the rest of the proof, we consider B ∈ [2 C ˜ η 2 , B ] .
Recall that I = (cid:110) 2 i T 3 : i ∈ { 0 , 1 , . . . , k } (cid:111) .
We denote by α min = 1 T 3 the minimum element of I , and α max = 2 k T 3 ≥ T the maximum element of I .
Denote by
[Equation]
As B ∈ [2 C ˜ η 2 , B ] , we have (cid:18) B [Equation] (cid:19)∈ ( α min , α max ] .
Indeed, (cid:18)
[Equation]
(cid:19)
[Equation]
≤
[Equation] Therefore [Equation] by the deﬁnition of i B
[Equation]
we have [Equation] Again by our assumption on B ,
[Equation]
[Equation]
Hence, the premises of Lemma 4 is satisﬁed for i
= i B ; this gives that with probability 1 − δ/ 2 ,
[Equation]
13) and
[Equation]
Now from Equation (14) and the deﬁnition of i B , we have
[Equation]
Applying Lemma 3, we deduce that for all t in [ T ] , A i B t = 1 .
Plugging this back to Equation (13), we have
[Equation]
where the second inequality is from the lower bound of α i B in Equation (12).
Combining the above observation with Lemma 1, along with the union bound, we get that with probability 1 − δ ,
[Equation]
[-Any online active learning algorithm A with label budget B has regret Ω(( (cid:80) mu =
-]
t =
(ˆ y t − (cid:104) θ ∗ , x t (cid:105) )
2 =
˜
O  
T (cid:88) t = ∆ t 
 =
˜ O (cid:32) ˜
η 2 ( (cid:80) u √ d u T u )
B (cid:33) .
A.3. Proof of Theorem
Recall that we deﬁne
[Equation]
and
[Equation]
Analogous to Theorem 1, the following theorem provides the query and regret guarantees of of Algorithm 3.
Theorem 5.
Suppose the example sequence { x t } T t =1 has the following structure: [ T ] has an admissible partition (cid:8)
I u : u ∈ [ m ] (cid:9) , where for each u , | I u | = T u , and the eluder dimension of F w.r.t.
{ x t } t ∈ I u is d (cid:48) u .
Suppose α ≥ ˜ η 2 max u ∈ [ m ] R u /T u .
With probability 1 − δ , Algorithm 3 satisﬁes:
Its query complexity Q = ˜
O (˜ η · √ α (cid:80) u √ R u T u ) .
Its regret R = ˜
O
(˜ η · (cid:80) u √ R u T u ) / √ α .
We shall prove Theorem 4 directly below; the proof of Theorem 5 follows as a corollary, using the same argument in the proof of Theorem 2; we note that the admissibility condition on domain partition { I u } mu =1 ensures that { A it ( r it − p it ) 1 [ t ∈ I ] } Tt = and { A it ( r it − p it )
∆ t 1 [ t ∈ I ] } Tt =1 are still martingale difference sequences in our proof.
Proof of Theorem 4.
We focus on proving the analogues of Lemma 1 and Lemma 2; the rest of proof follows the same argument as the proof of Theorem 2 and is therefore omitted.
Lemma 5 (Analogue of Lemma 1) .
With probability 1 − δ/ 2 , R ≤ (cid:80) Tt =1 ∆ t .
Proof.
Recall that the conﬁdence set at time t is F t = { f ∈ F : (cid:80) i ∈Q t ( f ( x i ) − ˆ f t ( x i )) 2 ≤ β |Q t | ( F , δ ) } .
By Russo &
Van Roy (2013, Proposition 2), we have that with probability 1 − δ/ 2 , f ∗ ∈ F t , for all t ∈
[ T ] .
Meanwhile, if f ∗ ∈ F t , for all t ∈ [ T ] , ( ˆ f t ( x t ) − f ∗ ( x t )) ≤ sup f 1 ,f 2 ∈F t ( f 1 ( x t ) − f 2 ( x t )) 2 = ∆ t .
This implies that the regret is bounded by R ≤ (cid:80)
Tt =1 ∆ t .
Lemma 6 (Analogue of Lemma 2) .
(cid:80) t ∈ I
u q t ∆ t ≤ R u .
Proof.
Let k
= | I u ∩ Q T | and write d = d (cid:48) u as a shorthand.
Let ( D 1 , . . .
, D k ) be { ∆ t : t ∈ I u
∩ Q T } sorted in non-increasing order.
We have (cid:88)
[Equation]
D j 1 [ D j ≤ 1 /T 4 ] + k (cid:88)
j
{+= D j 1 [ D j > 1 /T 4 ] . Clearly, (cid:80) kj =1 D j 
+}
[ D j ≤ 1 /T 4 ] ≤ T u T 2 .
We know for all j ∈ [ k ] , D j ≤ 4 .
In addition, D j > (cid:15) 2 ⇐⇒ (cid:80) t ∈ I u ∩Q T 1 [∆ t > (cid:15) 2 ] ≥ j .
By Lemma 7 below, this can only occur if j < (4 β T /(cid:15) 2 + 1) d .
Thus, when D j > (cid:15) 2 , j < (4 β T /(cid:15) 2 + 1) d , which implies (cid:15) 2 < 4 β T d j − d .
This shows that if
D j > 1 /T 4 , D j ≤ min (cid:110) 4 , 4 β T d j − d (cid:111) .
Therefore (cid:80) j D j
[ D j > 1 /T 4 ] ≤ 4 d + (cid:80) kj =
d +1 4 β T d j − d ≤ 4 d + 4 dβ T log T u .
Consequently, (cid:88)
[Equation]
The following lemma generalizes Russo & Van Roy (2013, Proposition 3), in that it considers a subsequence of examples coming from a subdomain of X .
We deﬁne dim EI as the eluder dimension of F with respect to support { x t : t ∈ I } .
It can be easily seen that dim EI u ≤ dim Eu .
Lemma 7.
Fix I ⊆
[ T ] .
If { β t ≥ 0 } Tt =1 is a nondecreasing sequence and F t :=
{ f ∈ F : (cid:80) i ∈Q t ( f ( x i ) − ˆ f t ( x i )) 2 ≤ β |Q t | ( F , δ ) } , then
[Equation]
Proof.
Let k = | I ∩ Q T | , ( a 1 , . .
. , a k )
=
( x t : t ∈ I ∩ Q T ) , and ( b 1 , . . .
, b k ) =
(∆ t : t ∈ I ∩ Q T ) .
First, we show that if b j > (cid:15) 2 then a j is (cid:15) -dependent on fewer than 4 β T /(cid:15) 2 disjoint subsequences of ( a 1 , . . .
, a j − 1 ) , for j ≤ k .
If b j > (cid:15)and a j = x t , there are f 1 , f 2 ∈ F t such that f 1 ( a j ) − f 2 ( a j ) > (cid:15) .
By deﬁnition, if a j is (cid:15) -dependent on a subsequence ( a i 1 , . . .
, a i p ) of ( a 1 , . . . , a j − 1 ) , then (cid:80) p l =1 ( f 1 ( a i l ) − f 2 ( a i l )) 2 > (cid:15) 2 .
Thus, if a j = x t is (cid:15) -dependent on K subsequences of ( a 1 , . . .
, a j − 1 ) , then (cid:80) i ∈Q t ( f 1 ( x i ) − f 2 ( x i ))
2 > K(cid:15) 2 .
By the triangle inequality, (cid:115) (cid:88) i ∈Q t ( f 1 ( x i ) − f 2 ( x i )) 2 ≤ (cid:115) (cid:88) i ∈Q t ( f 1 ( x i ) − f
[Equation]
Require: Hypotheses set F , time horizon T , label budget B , parameter δ , noise level η .
1: Labeled dataset Q ← ∅ .
2: Number of copies k ← 4 (cid:100) log 2 T (cid:101) .
3: for i = 0 to k do 4:
Parameter α
i ← 2 i /T
5: for t = 1 to T do 6:
Predict ˆ f t ← argmin f ∈F (cid:80) i ∈Q ( f ( x i ) − y i ) 2 .
Conﬁdence set F t ← { f ∈ F : (cid:80) i ∈Q ( f ( x i ) − ˆ f ( x i ))
2 ≤ β |Q| ( F , δ ) } , 8: where β k := 8 η 2 log (4 N ( F , 1 /T 2 , (cid:107) · (cid:107) ∞ ) /δ )
{++ 2 k/T 2 (16 + (cid:112) 
+}
η 2 ln
(16 k 2 /δ )) .
Uncertainty estimate ∆ t = sup f 1 ,f 2 ∈F t (cid:12)(cid:12) f 1 ( x t ) − f 2 ( x t ) (cid:12)(cid:12) 2 .
10: for i = 0 to k do 11: if (cid:80) t − 1 j =1 q ij < (cid:98) B/k (cid:99) then 12:
With probability min { 1 , α i ∆ t } , set q it = 1 .
13: if (cid:80) i q it > 0 then 14:
Query y t .
Q ← Q (cid:83) { t } .
Thus, K < 4 β T /(cid:15) 2 .
Next, we show that in any sequence of elements in I , ( c 1 , . . .
, c τ ) , there is some c j that is (cid:15) -dependent on at least τ/d −disjoint subsequences of ( c 1 , . . .
c j − 1 ) , where d :=
dim EI ( F , (cid:15) ) .
For any integer K satisfying Kd + 1 ≤ τ ≤ Kd + d , we will construct K disjoint subsequences C 1 , . . .
, C K .
First let C i =
( c i ) for i ∈
[ K ] .
If c K +1 is (cid:15) -dependent on C 1 , . . .
, C K , our claim is established.
Otherwise, select a C i such that c K +1 is (cid:15) -independent and append c K +1 to C i .
Repeat for all j > K + 1 until c j is (cid:15) -dependent on each subsequence or j = τ .
In the latter case (cid:80) | C i | ≥ Kd , and | C i | =
d .
In this case, c τ must be (cid:15) -dependent on each subsequence, by the deﬁnition of dim EI .
Now take ( c 1 , . . .
, c τ ) to be the subsequence ( a t 1 , . .
. , a t τ ) of ( a 1 , . .
. , a k ) consisting of elements a j for which b j > (cid:15) 2 .
We proved that each a t j is (cid:15) -dependent on fewer than 4 β T /(cid:15) 2 disjoint subsequences of ( a 1 , . . .
, a t j − 1 ) .
Thus, each c j is (cid:15) -dependent on fewer than 4 β T /(cid:15) 2 disjoint subsequences of ( c 1 , . . .
, c j − 1 ) .
Combining this with the fact that there is some c j that is (cid:15) -dependent on at least τ/d − 1 disjoint subsequences of ( c 1 , . .
c j − 1 ) , we have τ/d − 1 < 4 β T /(cid:15) 2 .
Thus, τ < (4 β T /(cid:15) 2 + 1) d .
A.4. Analysis of uniform query strategy for online active linear regression with oblivious adversary
Theorem 6.
With probability 1 − δ , the uniformly querying strategy with probability µ achieves E
[ R ] = ˜ O (cid:16) ˜
η 2 dµ (cid:17) and
E [ Q ] = µT .
Proof sketch.
As Q = (cid:80) Tt =1 q t is a sum of T iid Bernoulli random variables with means µ , E [ Q ] = µT . We now bound the regret of the algorithm.
We still deﬁne ∆ t = ˜
η 2 min { 1 , (cid:107) x t (cid:107) 2 M − 1 t } .
Using Lemma 2 with { a i } ki =1 = { x t } Tt =1 , and S = Q T , (cid:80) t q t ∆ t =
˜ O
(˜ η 2 d ) .
Let Z t =
q t ∆ t .
We have Z t ≤ ∆ t ≤ ˜ η 2 , E t − 1 Z t = µ ∆ t , and E t − 1 Z 2 t ≤ ˜ η 2 µ ∆ t .
Applying Freedman’s inequality, with probability 1 − δ/ 2 , T (cid:88)
[Equation]
The above inequality implies that (cid:80) Tt =1 ∆ t =
˜ O (cid:16) ˜
η 2 dµ (cid:17) .
Now, applying Lemma 1 and take the union bound, we have that with probability 1 − δ ,
[-+ 2 k/T 2 (16 + (cid:112) 
 η 2 ln (16 k 2 /δ )) . Uncertainty estimate ∆ t-]R = ˜
O  
T (cid:88) t = ∆ t 
 =
˜ O (cid:32) ˜
η 2 d µ (cid:33) .
Use the basic relationship between the expectation and tail probability E [ R ] = (cid:82) ∞ 0 P ( R ≥ a ) da , we conclude that E [ R ] = ˜ O (cid:16) ˜
η 2 dµ (cid:17) .
B. Proofs for lower bounds B.1. Proof of Theorem
We restate a slightly strengthened version of Theorem 3 here: the assumption B ≥ (cid:80) mu =1 √ d u T u · (cid:112) max u ∈ [ m ] d u /T u is weakened to B ≥ (cid:80) mu =1 d u .
Theorem 3.
For any η ≥ 1 , any set of positive integers (cid:8) ( d u , T u ) (cid:9) m u
=1 and integer B that satisfy d u ≤ T u , ∀ u ∈ [ m ] , m (cid:88) u
= d u ≤ d,
B ≥ m (cid:88) u
= d u , there exists an oblivious adversary such that:
It uses a ground truth linear predictor θ (cid:63) ∈ R
d such that (cid:107) θ ∗ (cid:107) 2 ≤ √ d , and (cid:12)(cid:12) (cid:104) θ ∗ , x t (cid:105) (cid:12)(cid:12) ≤ 1 ; in addition, the noises { ξ t } Tt = are sub-Gaussian with variance proxy η 2 . 2.
It shows example sequence { x t } Tt =1 such that [ T ] can be partitioned into m disjoint nonempty subsets { I u } mu =1 , where for each u , | I u | = T u , and { x t } t ∈ I u lie in a subspace of dimension d u . 3.
Any online active learning algorithm A with label budget B has regret Ω(( (cid:80) mu =1 √ d u T u )
/B ) .
Proof.
Our proof is inspired by Vovk (2001, Theorem 2).
For u ∈ [ m ] and i ∈ [ d u ] , deﬁne c u,i =
e (cid:80) u − 1 v =1 d v + i , where e j denotes the j -th standard basis of R d .
It can be easily seen that all c u,i ’s are orthonormal.
In addition, for a vector θ ∈ R d , denote by θ u,i
= θ (cid:80) u − 1 v =1 d v + i .
For task u , we construct domain X u = span( c u,i : i ∈
[ d u ]) .
The sequence of examples shown by the adversary is the following: it is divided to m blocks, where the u -th block occupies a time interval I u
= [ (cid:80) u − 1 v =1 T v + 1 , (cid:80) uv =1 T v ] ;
Each block is further divided to d u subblocks, where for i ∈ [ d u − 1] , subblock ( u, i ) spans time interval I u,i =
[ (cid:80) u − 1 v =1 T v + ( i − 1) (cid:98) T u /d u (cid:99) + 1 , (cid:80) u − 1 v =1 T v
+ i (cid:98) T u /d u (cid:99) ] , and subblock ( u, d u ) spans time interval I u,d u =
[ (cid:80) u − 1 v =1 T v + ( d u − 1) (cid:98) T u /d u (cid:99)
+ 1 , (cid:80) u − 1 v =1 T v + T u ] .
At block u , examples from domain X u are shown; furthermore, for every t in I u,i , i.e. in the ( u, i ) -th subblock, example c u,i is repeatedly shown to the learner.
Observe that ( u, i ) -th subblock contains at least (cid:98) T u d u (cid:99) ≥ T u 2 d u examples, as T u ≥ d u .
We ﬁrst choose θ ∗ from distribution D θ , such that for every coordinate j ∈ [ d ] , θ ∗ i ∼ Beta(1 , 1) , which is also the uniform distribution over [0 , 1] .
Given θ ∗ , the adversary reveals labels using the following mechanism: given x t , it draws y t ∼ Bernoulli( (cid:104) θ ∗ , x t (cid:105) )
independently and optionally reveals it to the learner upon learner’s query.
Speciﬁcally, given θ ∗ , if t ∈ I u,i , y t ∼ Bernoulli
( θ ∗ u,i ) .
By Hoeffding’s Lemma, ξ t = y t − θ ∗ u,i is zero mean subgaussian with variance proxy 14 ≤ η 2 .
Denote by N u,i ( t ) =
(cid:80) s
∈
I u,i : s ≤ t q s the number of label queries of the learner in domain ( u, i ) up to time t .
Because the learner satisﬁes a budget constraint of B under all environments, we have
[Equation]
Adding 2 (cid:80) mu =1 d u on both sides and by linearity of expectation, we get
[Equation]
On the other hand, we observe that the expected regret of the algorithm can be written as follows:
[Equation]
where the expectation is with respect to both the choice of θ ∗ and the random choices of A .
We deﬁne a ﬁltration {F t } Tt =1 , where F t is the σ -algebra generated by (cid:8) ( x s , q s , y s q s ) (cid:9) t s =1 , which encodes the informative available to the learner up to time step t .
We note that ˆ y t is F t − 1 -measurable.
Denote by N + u,i ( t ) =
(cid:80) s
∈
I u,i : s ≤ t q s · 1 ( y s = 1) , which is the number of 1 labels seen on example c u,i by the learner up to round t − 1 .
Observe that both
N + u,i ( t − 1) and N u,i ( t − 1) are F t − 1 -measurable.
Observe that conditioned on the interaction logs ( x s , q s ,
y s q s ) t − 1 s =1 , the posterior distribution of θ ∗ u,i is Beta(1 + N + u,i ( t − 1) , 1 + N u,i ( t − 1) − N + u,i ( t − 1)) .
Therefore, deﬁne random variable ˆ y ∗ t = E (cid:104) θ ∗ u,i | F t − 1 (cid:105) =
1+ N + u,i 2+ N u,i , we have by bias-variance decomposition,
[Equation]
Summing over all time steps, we have
[Equation]
On the other hand, from Lemma 8, we have for all t ∈ I u,i ,
[Equation]
where f ( γ ) =
min( γ · (1 − γ ) , (2 γ − 1) 2 ) .
By the tower property of conditional expectation and conditional Jensen’s inequality, we have
[Equation]
Summing over all t in I u,i , and then summing over all subblocks ( u, i ) : u ∈ [ m ] ,
i ∈
[ d u ] , and using the aforementioned fact that the ( u, i ) subblock has at least T u 2 d u examples, we have
[Equation]
[Equation]
Combining the above inequality with Equation (15), we have:
[Equation]
where the second inequality is from Cauchy-Schwarz.
Now taking expectation over θ
[Equation]
using Jensen’s inequality and
Lemma 9 that E (cid:113) f
( θ ∗ u,i ) ≥ 125 , and some algebra yields
[Equation]
In conclusion
[Equation]
we have
[Equation]
As the above expectation is over θ ∗ chosen randomly from D θ , there must exists an θ ∗ from supp( D θ ) =
[0 , 1] d such that
[Equation]
holds.
This θ ∗ has (cid:96) 2 norm at most (cid:113)(cid:80) dj
=1 ( θ ∗ j ) 2 ≤ √ d . Lemma 8. If t is in I u,i , then [Equation]
where f ( γ ) =
min (cid:0) γ (1 − γ ) , (2 γ − 1) 2 (cid:1) . Proof. We condition on N u,i ( T ) = m , and a value of θ ∗ . Recall that ˆ y ∗ t = 1+ N + u,i 2+ N u,i = 1+ N + u,i 2+ m , where N + u,i can be seen as drawn from the binomial distribution
Bin( m, θ ∗ u,i ) . [Equation] Lemma 9. Suppose Z ∼ Beta(1 , 1) .
Then E (cid:104)(cid:112) f ( Z ) (cid:105) ≥ 125 . Proof. We observe that
[Equation]
Now, for all z ∈ [ 15 , 25 ] , (cid:112) f ( z ) ≥ (cid:113) 125 = 15 , which implies that the above integral is at least 125 .
B.2. Lower bound for unstructured domains
We have the following lower bound in the case when there is no domain structure.
Theorem 7.
For any set of positive integers d, T, B such that d ≤ T and d ≤ B , there exists an oblivious adversary such that:
it uses a ground truth linear predictor θ (cid:63) ∈ R d such that (cid:107) θ ∗ (cid:107) 2 ≤ √ d , and (cid:12)(cid:12) (cid:104) θ ∗ , x t (cid:105) (cid:12)(cid:12) ≤ 1 . 2.
any online active learning algorithm A with label budget B has regret at least Ω (cid:16) dTB (cid:17) .
Proof.
This is an immediate consequence of Theorem 3, by setting m = 1 , d 1 = d , T 1 = T , and the label budget equal to
B .
C. The c-cost model for online active learning
We consider the following variant of our learning model, which models settings where the cost ratio between a unit of square loss regret and a label query is c to 1.
In this setting, the interaction protocol between the learner and the environment remains the same, with the goal of the learner modiﬁed to minimizing the total cost, formally C = cR + Q .
We call the above model the c -cost model .
We will show that Algorithm 1 achieves optimal cost up to constant factors, for a wide range of values of η and c .
Theorem 8.
For any η ≥ 1 , set of positive integers (cid:8) ( d u , T u ) (cid:9) m u =1 such that d u ≤ T u , ∀ u ∈ [ m ] , (cid:80) mu =1 d u ≤ d , cost ratio c ≥ max u d u T u , there exists an oblivious adversary such that:
it uses a ground truth linear predictor θ (cid:63) ∈ R d such that (cid:107) θ ∗ (cid:107) 2 ≤ √ d , and (cid:12)(cid:12) (cid:104) θ ∗ , x t (cid:105) (cid:12)(cid:12) ≤ 1 ; in addition, the subgaussian variance proxy of noise is η 2 . 2.
it shows example sequence { x t } Tt =1 such that [ T ] can be partitioned into m disjoint nonempty subsets { I u } mu =1 , where for each u , | I u | = T u , and { x t } t ∈ I u lie in a subspace of dimension d u . 3.
any online active learning algorithm A has total cost Ω (cid:16) √ c · ( (cid:80) mu =1 √ d u T u ) (cid:17) .
Proof.
Consider any algorithm A .
Same as in the proof of Theorem 3, we will choose θ ∗ randomly where each of its coordinates is drawn independently from the Beta(1 , 1) distribution, and show the exact same sequence of instances { x t } Tt = and reveals the labels the same say as in that proof.
It can be seen that the η t ’s are subgaussian with variance proxy 1 , which is also subgaussian with variance proxy η 2 .
As A can behave differently under different environments, we deﬁne E (cid:2) Q | θ ∗ (cid:3) as A ’s query complexity conditioned on the adversary choosing ground truth linear predictor θ ∗ .
We conduct a case analysis on the random variable E (cid:2) Q | θ ∗ (cid:3) :
If there exists some θ ∗ ∈ [0 , 1] d , E (cid:2) Q | θ ∗ (cid:3) ≥ √ c (cid:16)(cid:80) mu =1 √ d u T u (cid:17) , then we are done: under the environment where the ground truth linear predictor is θ ∗ , the total cost of A , E (cid:2) C | θ ∗ (cid:3) , is clearly at least E (cid:2) Q | θ ∗ (cid:3) ≥
Ω (cid:18) √ c (cid:16)(cid:80) mu =1 √ d u T u (cid:17)(cid:19) .
If for every θ ∗ ∈ [0 , 1] d , E (cid:2) Q | θ ∗ (cid:3) ≤ √ c (cid:16)(cid:80) mu =1 √ d u T u (cid:17) , A can be viewed as an algorithm with label budget
B = √ c (cid:16)(cid:80) mu =1 √ d u T u (cid:17) .
By the premise that c ≥ max u d u T u , we get that B ≥ (cid:80) mu =1 √ d u T u · (cid:113)
d u T u
= (cid:80) mu =1 d u .
Therefore, from the proof of Theorem 3, we get that there exists a θ ∗ in [0 , 1] d , such that
[Equation]
which implies that the total cost of A , under the environment where the ground truth linear predictor is θ ∗ , E (cid:2) C | θ ∗ [Equation] , is at least
[Equation]
In summary, in both cases, there is an oblivious adversary that uses θ ∗ in [0 , 1] d , under which A has a expected cost of Ω
[Equation]
√ c
[Equation]
In the theorem below, we discuss the optimality of Algorithm 1 in the c -cost for a range of problem parameters.
Theorem 9.
Suppose η ∈ [1 , O (1)] ; in addition, consider a set of (cid:8) ( T u , d u ) (cid:9) m u =1 , such that min u T u /d u ≥ η .
Fix c ∈ [max u d u T u , 1 η 2 min
u T u d u ] .
We have
Under all environments with domain dimension and duration (cid:8) ( T u , d u ) (cid:9) m u =1 , such that (cid:107) θ ∗ (cid:107) ≤ C and max t ∈ [ T ] (cid:12)(cid:12) (cid:104) θ ∗ , x t (cid:105) (cid:12)(cid:12) ≤ 1 , QuFUR ( c ) (with the knowledge of norm bound C ) has the guarantee that
[Equation]
For any algorithm, there exists an environment with domain dimension and duration (cid:8) ( T u , d u ) (cid:9) m u =1 such that (cid:107) θ ∗ (cid:107) ≤ √ d and max t ∈ [ T ] (cid:12)(cid:12) (cid:104) θ ∗ , x t (cid:105) (cid:12)(cid:12) ≤ 1 , under which the algorithm must have the following cost lower bound:
[Equation]
Proof.
We show the two items respectively:
As c ≤ ˜ η 2 min u T u d u , and c ≥ max u d u T u ≥ 1˜ η 2 ( 1 ( (cid:80) u √ d u T u ) 2 ) , applying Theorem 1, we have that QuFUR ( c ) achieves the following regret and query complexity guarantees:
[Equation]
This implies that
[Equation]
By the condition that c ≥ max u d u T u , applying Theorem 8, we get the item.
D. The regret deﬁnition
Recall that in the main text, we deﬁne the regret of an algorithm as R =
(cid:80) Tt
=1 (ˆ y t − f ∗ ( x t )) 2 .
This is different from the usual deﬁnition of regret in online learning, which measures the difference between the loss of the learner and that of the predictor θ ∗ : Reg = {+(cid:80) Tt =
+}
(ˆ y t − y t ) 2 − (cid:80) Tt =1 ( f ∗ ( x t ) − y t ) 2 .
We show a standard a result in this section that the expectation of these two notions coincide.
Theorem 10.
E
[ R ] =
E [Reg] .
Proof.
Denote by F t − 1 be the σ -algebra generated by all observations up to time t − 1 , and x t .
As a shorthand, denote by E t − 1 [ · ] =
E [ · | F t − 1 ] .
Let Z t =
(ˆ y t − y t ) 2 − ( f ∗ ( x t ) − y t ) 2 ; we have
[Equation]
where the last inequality uses the fact that E t − 1 ( f ∗ ( x t ) − y t )
=
and ˆ y t − f ∗ ( x t ) is F t − 1 -measurable.
Consequently, E
Z t = E ( f ∗ ( x t ) − ˆ y t ) 2 .
The theorem is concluded by summing over all time steps t from 1 to T .
E. Online to batch conversion
In this section we show that by an standard application of online to batch conversion (Cesa-Bianchi et al., 2004a) on QuFUR, we obtain new results on active linear regression under the batch learning setting.
First we recall a standard result on online to batch conversion; for completeness we provide its proof here.
Theorem 11.
Suppose online active learning algorithm
A sequentially receives a set of iid examples ( x t , y t )
Tt =1 drawn from D , and at every time step t , it outputs predictor ˆ f t : X → Y .
In addition, suppose (cid:96) :
Y × Y → R is a loss function.
Deﬁne regret Reg = (cid:80) Tt =1 (cid:96) ( ˆ f t ( x t ) , y t ) − (cid:80) Tt =1 (cid:96) ( f ∗ ( x t ) , y t ) , and deﬁne (cid:96) D ( f ) =
E ( x,y ) ∼ D (cid:96) ( f ( x ) , y ) .
If E [Reg] ≤ R 0 , then,
[Equation]
E f ∼ uniform( ˆ f 1 ,..., ˆ f T ) (cid:96) D ( f ) (cid:105) − (cid:96) D ( f ∗ ) ≤ R 0 T .
Proof.
As Reg = (cid:80) Tt =1 (cid:96) ( ˆ f t ( x t ) , y t ) − (cid:80) Tt =1 (cid:96) ( f ∗ ( x t ) , y t ) , We have
R 0 ≥ E [Reg] =
T (cid:88) t =
E (cid:104) (cid:96) D
( ˆ f t ) (cid:105) − E  
T (cid:88) t = (cid:96) ( f ∗ ( x t ) , y t ) 
 = T ·   1 T
T (cid:88) t = E (cid:104) (cid:96) D
( ˆ f t ) (cid:105) − E ( x,y ∼ D (cid:96) ( f ∗ ( x ) , y ) .


The theorem is proved by dividing both sides by T and recognizing that
[Equation]
Combining Theorem 11 with Theorem 2, we have the following adaptive excess loss guarantee of Fixed-Budget QuFUR (Algorithm 2) when run on iid data with hidden domain structure.
Theorem 12.
Suppose the unlabeled data distribution D X is a mixture distribution: D X = (cid:80) mu =1 p u D u , where D u is a distribution supported on a subspace of R d of dimension d u and is a subset of (cid:110) x : (cid:107) x (cid:107) 2 ≤ 1 , (cid:12)(cid:12) (cid:104) θ ∗ , x (cid:105) (cid:12)(cid:12) ≤ 1 (cid:111) .
The conditional distribution of y given x is y =
(cid:104) θ ∗ , x (cid:105)
+ ξ where ξ is a subgaussian with variance proxy η 2 .
In addition, suppose we are given integer B , T 0 such that T 0 ≥ Ω
[Equation]
If Algorithm 2 is given dimension d , time horizon T ≥ T 0 , label budget B , norm bound C , noise level η as input, then:
It uses T unlabeled examples.
Its query complexity Q is at most B . 3.
Denote by (cid:96) (ˆ y, y )
=
(ˆ y − y )
the square loss.
We have,
[Equation]
Proof sketch.
From Theorem 11 it sufﬁces to show that
[Equation]
By Theorem 10, E [Reg] =
E
[ R ] , it therefore sufﬁces to show that
[Equation]
We ﬁrst show a high probability upper bound of R .
Given a sequence of unlabeled examples { x t } Tt =1 , we denote by S u the subset of examples drawn from component D u , and denote by T u the size of S u .
From the assumption of D u , we know that
S u all lies in a subspace of dimension d u .
Deﬁne event E as follows:
[Equation]
From the assumption that T ≥ T 0 ≥ Ω( ln m min u p u ) , we have that by Chernoff bound and union bound, P ( E ) ≥ 1 − 1 T 2 . Conditioned on event E happening, we have that by the assumption that T ≥ T 0 ≥
[Equation]
Therefore, applying Theorem 2, we have that conditioned on event E happening, with probability 1 − 1 T 2 over the draw of { y t } Tt =1 ,
[Equation]
Combining the above two equations and using union bound, we conclude that with probability 1 − 2 T 2 ,
[Equation]
Observe that with probability 1 , ˆ y t ∈ [ − 1 , 1] and (cid:104) θ ∗ , x t (cid:105) ∈ [ − 1 , 1] .
Therefore, R = (cid:80) Tt
=1 (ˆ y t − (cid:104) θ ∗ , x t (cid:105) ) 2 ∈ [0 , 4 T ] .
Hence,
[Equation]
The theorem follows.