Efficient and Near-Optimal Smoothed Online Learning for Generalized Linear Functions
Abstract
Due to the drastic gap in complexity between sequential and batch statistical learning, recent work has studied a smoothed sequential learning setting, where Nature is constrained to select contexts with density bounded by 1 /σ with respect to a known measure µ .
Unfortunately, for some function classes, there is an exponential gap between the statistically optimal regret and that which can be achieved efficiently.
In this paper, we give a computationally efficient algorithm that is the first to enjoy the statistically optimal log( T/σ ) regret for realizable K -wise linear classification.
We extend our results to settings where the true classifier is linear in an over-parameterized polynomial featurization of the contexts, as well as to a realizable piecewise-regression setting assuming access to an appropriate
ERM oracle.
Somewhat surprisingly, standard disagreement-based analyses are insufficient to achieve regret logarithmic in 1 /σ .
Instead, we develop a novel characterization of the geometry of the disagreement region induced by generalized linear classifiers.
Along the way, we develop numerous technical tools of independent interest, including a general anti-concentration bound for the determinant of certain matrix averages.
Introduction
In batch statistical learning, a learner faces a set of independent examples drawn from a given distribution and is tasked with generalizing to novel examples drawn from that same distribution.
In sequential or online learning, however Nature may adversarially select examples to thwart the learner’s progress and success is defined only in comparison to the best a priori predictor.
Due to the wide range of application and minimal set of assumptions, online learning has received considerable recent attention.
For concreteness, consider binary classification, where a sequence of T examples takes the form ( x t , y t ) ∈ R d × {− +1 } .
Even in the realizable setting , where there exists a true f ⋆ in a pre-specified class of functions F for which f ⋆ ( x t ) =
y t for all t ∈ { 1 , 2 , . . .
, T } , the gap between batch and statistical learning and sequential learning can be drastic: when d = 1 , the class of linear thresholds f θ ( x ) =
sign( x − θ ) has VC dimension one and is thus learnable in the PAC framework
[Wainwright, 2019].
A sequential adversary, however, can select x t so as to force the linear to misclassify Ω( T ) points [Littlestone, 1988].
To circumvent the pessimism of the sequential setting, recent works [Rakhlin et al., 2011, Haghtalab et al., 2020, 2021, Block et al., 2022, Haghtalab et al., 2022] have studied the smoothed sequential learning paradigm, where the adversary is constrained to choose x t at random from any probability distribution p t with density at most 1 /σ with respect to a known measure µ .
The most current of these results point to a striking statistical computational gap: whereas there exist algorithms which attain regret that scales with (cid:112) T log( /σ ) , computationally efficient algorithms can only hope for poly( T/σ ) regret in general, even against a realizable adversary [Haghtalab et al., 2022,
Theorem 5.2].
In many d -dimensional settings, natural choices of µ yield σ = exp( − Ω( d )) , and thus the exponential separation in σ translates into an exponential separation in dimension.
This gap motivates the following question: can the statistical-computational gap be eliminated in more structured settings?
In this work, we answer the question affirmatively for a variety of natural function classes.
A better understanding of what function classes allow computationally efficient, statistically optimal regret-minimizing algorithms remains a promising direction for future research.
Contributions.
We show that for certain classes of realizable smoothed online classification problems, there exists a computationally efficient algorithm which enjoys the statistically optimal log( T/σ ) regret scaling, when the base measure µ is uniform on the unit-ball.
Specifically, we provide computationally efficient algoirthms for achieving the statistically optimal regret bound for the following function classes:
• For affine thresholds, • For affine thresholds in nonlinear features, • For K -class affine classification, • For piecewise affine regression
We also provide lower bounds that demonstrate the statistical optimality of our algorithms.
Furthermore, we apply our results to noiseless contextual bandits and get a fast algorithm that achieves optimal regret dependence on the horizon, up to logarithmic factors.
Finally, we present a complementary approach based on the perceptron algorithm which is robust to adversarial corruptions of the labels y t , and enjoys a polynomial regret in a “directional smoothness” parameter which interpolates between the log(1 /σ )
-guarantees attained above in the realizable setting, and the poly(1 /σ )
bounds from prior work.
We emphasize that, though we adopt the smoothed online learning setting of
Rakhlin et al.
[2011], Haghtalab et al.
[2021], Block et al.
[2022], we use entirely different techniques involving Ville’s inequality [Ville, 1939], geometric measure theory, and convex geometry.
Moreover, in none of these works was the question of adapting to realizability explored; thus, we provide the first regret bounds that are logarithmic in both the horizon and the smoothness parameter.
We now discuss some related work:
Online Learning.
Extensions of classical learning theory to the online setting have proliferated due to the scope of application.
Several works [Littlestone, 1988, Blumer et al., 1989, Ben-David et al., 2009, Rakhlin et al., 2015a] have explored the gap in statistical rates between classical and online learning settings, with Littlestone [1988], Blumer et al.
[1989] showing that the class of one dimensional thresholds, which is easy to learn in the batch setting, is not learnable with adversarial data.
Other works, such as Rakhlin et al.
[2015a], Rakhlin and Sridharan [2013], Rakhlin et al.
[2015b], Block et al.
[2021], Rakhlin and Sridharan [2014] have provided sequential analogues of classical notions of complexity that characterize minimax regret, as well as providing computational separation between classical and online learning [Hazan and Koren, 2016].
Due to the statistical and computational hardness results presented in the aforementioned work, there has been great interest in finding realistic, robust assumptions, such as smoothness, that allow for efficient learning.
Smoothed Online Learning.
Smoothed analysis was first proposed in Spielman and Teng [2004] as a way to explain the success of the simplex algorithm of Klee and Minty [1972] by combining the polynomial time bounds of an average-case analysis with the verisimilitude of a worst-case analysis.
Since then, smoothed analysis has been applied to explain the empirical success of many algorithms [Roughgarden, 2021].
In the learning setting, Rakhlin et al.
[2011] proposed smoothed adversaries and proved regret bounds for linear thresholds in R d ; their proof, however, was nonconstructive and did not achieve logarithmic regret in the realizable setting.
The use of smoothed adversaries was essential due to the hardness results discussed above.
In a series of works Haghtalab et al.
[2020, 2021] generalized Rakhlin et al.
[2011] and showed the regret depending on the VC dimension was possible in the smoothed online learning setting, albeit with computationally inefficient algorithms.
Recently, Block et al.
[2022], Haghtalab et al.
[2022] generalized Haghtalab et al.
[2021] to allow for continuous labels and, more importantly, provided oracle-efficient algorithms for achieving vanishing regret in the smoothed setting.
These papers also showed that the dependence on σ in the regret bounds of their oracle-efficient algorithms, which was polynomial, could not in general be reduced to the logarithmic dependence achievable by the inefficient algorithms, thereby exposing a statistical-computational gap.
Unlike other recent works such as Block et al.
[2022], Haghtalab et al.
[2022], we do not use the coupling approach [Haghtalab et al., 2021] to prove our regret bounds.
Classification with Linear Thresholds.
Considering the ubiquity of linear thresholds in classification, the list of relevant references is far too long to include here; as such, we highlight only those most germane to our work.
The perceptron algorithm was introduced in Rosenblatt [1958] and a margin-based mistake bound was proved in Novikoff [1963].
There have been many variations on and applications of this bound, from Ben-David et al.
[2009] using it to bound the Littlestone dimension of linear thresholds with margin to dealing with non-realizable samples [Crammer et al., 2006, Freund and Schapire, 1999].
To the best of our knowledge our work constitutes the first to explore the effect that a smoothed adversary has on the perceptron algorithm.
Disagreement Coefficient and Active Learning.
Intuitively, our analysis is similar to works in active learning based on the disagreement coefficient [Hanneke, 2007, 2011, Hanneke et al., 2014,
Wang, 2011].
Indeed, as we shall see, our regret bounds arise by bounding the probability that a point falls into the disagreement region in a similar way as, for example, Hanneke [2007] controls the label complexity of active learning.
We will note in Remark 3, however, that an approach grounded purely in the disagreement coefficient cannot hope to achieve regret logarithmic in σ in the smoothed setting.
Indeed, our approach incorporates a finer understanding of the geometry, accomodated by the more limited scope of application of our techniques, which allows us to prove tight rates.
In Section 2, we setup the learning problem and introduce some necessary notions from convex geometry, as well as fixing notation.
In Section 3, we highlight two technical results that form the foundation of our approach, before, as a warmup, applying them to the case of classification with linear thresholds in Section 4.
In Section 5, we generalize beyond linear thresholds to allow for offset and nonlinear features.
Finally, in Section 6, we move beyond binary classification by extending our results to K -class affine classification, piecewise affine regression, and noiseless contextual bandits.
Preliminaries
In this section, we provide basic definitions and setup the learning problem.
We begin by defining a smooth distribution, as in Block et al.
[2022], Haghtalab et al.
Definition 1.
Let µ be a probability measure on a measurable space X .
For some 0 < σ ≤ 1 , we say that a measure p on X is σ -smooth with respect to µ if the likelihood d p d µ ≤ 1 σ is uniformly bounded.
We consider the smoothed online learning setting.
First, a horizon T ∈ N is fixed and a distribution µ on X is chosen.
For each step 1 ≤ t ≤ T , Nature chooses a distribution p t , possibly depending on the history, such that p t is σ -smooth with respect to µ and samples x t ∼ p t as well as choosing some y t ∈ Y .
The learner sees x t , chooses ˆ y t and suffers loss ℓ (ˆ y t , y t ) .
Given a function class F of functions mapping X → Y , the learner attempts to minimize regret, where regret is defined as:
[Equation]
In the sequel, for the sake of simplicity, we take X = B
d 1 to be the unit ball, µ = µ d to be the uniform measure on B d 1 , and ℓ (ˆ y t , y t ) =
I (ˆ y t ̸ = y t ) to be the 0-1 loss.
Remark 1 (Scaling of σ ) .
A natural example of a smoothed adversary is one that is allowed to place (cid:98) x t in a worst-case manner, which gets perturbed by some small additive noise, chosen uniform on ε · B d 1 , to become x t ; this adversary is σ = ε d smooth.
For such situations, polynomial dependence on σ in the regret translates into something exponential in dimension.
Remark 2 (Other measures µ ) .
Assuming the dominating measure µ = µ d is not overly strong: if µ is another measure on B d 1 for which d µ d µ d ≥ c > 0 , then, because our regret bounds are logarithmic in σ , our results will still hold with an additive term of log (cid:0) 1 c (cid:1) .
For much of the paper, we assume that Nature is realizable with respect to F , i.e., for some f ⋆ ∈ F , f ⋆ ( x t ) =
y t for all 1 ≤ t ≤ T .
In this case, Reg T is just a mistake bound: Reg T = (cid:80)
Tt
=
I { ˆ y t ̸ =
y t } .
The foundations of our analysis consider the class of linear threshold classifiers
[Equation]
We identify F d lin with the set of w ’s defining it, so that we may treat it as, itself, a subset of B d 1 ; other function classes are similarly identified with their parameters (without further comment).
At the core of our base algorithm is the computation of the John ellipsoid [John, 1948, Ball et al., 1997], the maximal volume ellipsoid contained in a convex body.
1 It is well-known that given a polytope in R d , the John ellipsoid can be computed in time polynomial in d and the number of faces [Boyd and Vandenberghe, 2004].
In particular, we compute the John ellipsoid of the version space , F t , where for any time t , we let F t = { f ∈ F d lin | f ( x s ) =
y s for all s < t } , which is a polytope with t ≤ T faces.
An important concept in our analysis is the notion of Hausdorff measure , which generalizes the standard notions of volume and surface area in R d ; we will denote the k -dimensional
Hausdorff measure (see Definition 19) by vol k ( · ) .
More detail on both the John ellipsoid and the
Hausdorff measure can be found in Appendix B.
Notation.
For a set U ⊂ R d , we denote by ∂ U its boundary.
We let B dr denote the ball of radius r around the origin in R d and let S d − 1 = ∂B d 1 . Letting Γ denote the Γ -function, let ω d
= π d [Equation] Γ( d/ 2+1) denote the volume of B d 1 and let µ d denote the uniform measure on B d 1 normalized to be a probability measure.
If ϕ : R n → R m is Lipschitz, we denote the Jacobian by Dϕ .
Lastly, we use “ ≲ ” to denote inequality up to universal, problem-independent constants.
The Technical Workhorses
In this section we introduce the two key workhorse results that provide the technical foundation for the rest of the paper.
The first result is a purely probabilistic statement that we use as a blackbox throughout the paper to turn probabilistic and geometric theorems into regret bounds in the realizable, smoothed online learning setting.
The second result is a geometric statement that allows us to apply the black box regret bound to the case of classification with affine thresholds.
3.1 An Abstract Decay Analysis
We begin with an abstract, technical result that will form the basis for all of our regret bounds.
We first introduce the following definition:
Definition 2.
Let µ be a measure on some set Z and let ℓ t : Z → { 0 , 1 } be a sequence of loss functions.
For R > 0 and 0 < c < 1 , we say that the sequence ( ℓ t , z t ) satisfies ( R, c ) -geometric decay with respect to µ if there exists a sequence of nonnegative numbers R t with R 1 =
R satisfying the following two properties:
For all t , µ ( { z : ℓ t ( z ) = 1 } ) ≤ R t .
For any t such that ℓ t ( z t ) = 1 , we have R t +1 ≤ cR t .
To motivate this admittedly abstract definition, consider the case of online classification with thresholds f θ ( x ) = sign( x − θ ) from the introduction, with µ uniform on [0 , 1] × {± 1 } (note that this does not precisely fit into the linear setting described above due to the offset); take z t =
( x t , y t ) and ℓ t ( z t ) =
I [ˆ y T ̸ =
y t ] , where the learner predicts ˆ y t at each time t .
By realizability, ℓ t ( z ) = 1 only when x t falls in the “region of disagreement,”i.e.
the interval the rightmost x s labelled − 1 and the leftmost x s labelled 1 .
To see why this is true, note that the “version space,” i.e., the set of thresholds that correctly classify all the data so far, is exactly this interval; for us to make a mistake, there must be two functions in the version space that disagree on x t , which can only happen if x t itself is in the version space.
If the learner denotes by w t the midpoint of the region of disagreement, then any mistake forces the version space, and thus the disagreement region, to shrink by a factor of 2.
We see then that ( ℓ t , z t ) satisfy (cid:0) 1 , 12 (cid:1) -geometric decay with respect to the uniform measure.
If the adversary were constrained to choose x t ∼ µ at each time step, it is intuitive that we should not expect many mistakes to be made because, after any mistake, the probability that we make a mistake in some future interval decreases.
In the following result, we show that this intuition holds in the more general smoothed setting:
Lemma 3 (Abstract Decay Lemma) .
Suppose that a sequence ( ℓ t , z t ) satisfies ( R, c ) -geometric decay with respect to some µ on Z , and that for all t , there is some p t that is σ -smooth with respect to µ and z t ∼ p t .
Then for all T ∈ N , with probability at least 2 1 − δ ,
[Equation]
Proof Sketch.
We break our analysis into epochs whose lengths h m are tuned at the end of the proof.
We then consider a sequence of stopping times τ m that count the number of epochs of length
h m we experience in between the ( m − 1) st and m th time that ℓ t = 1 .
We then show that if h m is not too large relative to the Probability that ℓ t = 1 , then τ m − τ m − 1 is large with high probability and apply Ville’s inequality [Ville, 1939] to conclude that if m T is the maximal epoch-index m such that τ m ≤ T , then m T cannot be too large.
We again apply Ville’s inequality to show that if h m is not too large then the probability of multiple mistakes per epoch is small.
Because of the geometric decay property, the probability that ℓ t = 1 decreases exponentially in the number of mistakes and thus we may let h m grow exponentially in m and still not be too large to apply the above argument.
We then conclude by noting that if h m are growing exponentially in m then m T has to be logarithmic in T .
The details can be found in Appendix C.1.
■
If we return to the above example of online classification with thresholds, we see that Lemmaimmediately yields the first regret bound for realizable, smoothed online learning with thresholds that is logarithmic both in the horizon T and the smoothness parameter σ .
The intuition gleaned from one-dimensional thresholds that geometric decay suffices to ensure logarithmic regret will be key to the more general regret bounds we exhibit below.
3.2 A Volumetric Lemma
In the previous section, we saw that in the setting of realizable, smoothed online classification with one-dimensional thresholds, the learner can force the indicator of a mistake at time t to satisfy geometric decay; our second workhorse result will allow us to extend this fact to higher dimensions.
In the case of thresholds in the unit interval, the key intuition leading to geometric decay was the fact that the disagreement region was exactly the version space and thus shrinking the version space tautologically shrank the disagreement region as well.
In higher dimensions the situation is significantly more complicated.
We have the following result:
Lemma 4.
Let x 1 , . . .
, x t ∈ B d 1 and suppose that y 1 , . .
. , y t are realizable with respect to F d lin .
Define the disagreement region
[Equation]
where F t is the version space, defined in Section 2.
Then, recalling that ∂ F t is the boundary of F t ,
[Equation]
Note that by controlling the size of D t by that of F t , Lemma 4 is a direct generalization of the one-dimensional case; however, in contradistinction to that setting, the proof is much more difficult and the bound includes an extra term corresponding to the surface area of F t , which is unavoidable in general.
The full proof is in Appendix C.2, but we summarize the key points here.
Though the conclusion of Lemma 4 is intuitive, it requires significant technical effort to prove.
Proof Sketch of Lemma 4.
We first note that D t is contained in the set of points x such that there is some w ∈ F t with ⟨ w, x ⟩ = 0 ; thus the conclusion of Lemma 4 reduces to a geometric statement about the volume of the set of points orthogonal to at least one point in a given set can be.
It may seem like this should “obviously” be the volume of a ( d − 1) -dimensional ball multiplied by the
1: Initialize W 1 =
B
d 1 , w 1 = e2: for t = 1 , 2 , . . .
do 3:
Recieve x t , and predict ˆ y t
= sign( ⟨ w t , x t ⟩ ) , ( % self .
classify ( x t ) ) 4:
Update W t
+1 =
W t ∩ { w ∈ B d 1 | ⟨ w, x t y t ⟩ ≥ 0 } 5: if ˆ y t ̸ =
y t then ( % self .
errorUpdate ( x t ) ) 6: w t +1 ← JohnEllpsoidCenter ( W t +1 ) 7: % returns center of John Ellpsoid of given convex body volume of F t , but this is false: if F t is the equator of the sphere S d − 1 , then µ d ( F t ) = 0 , but the set of points orthogonal to at least one point in F t is the entirety of B d 1 .
Ruling out this pathology requires several steps, including a covering argument to reduce to the case where F t is a ball, and application of (a generalized) Steiner’s formula, and a deep geometric fact called Weyl’s Tube Formula [Weyl, 1939, Gray, 2003] that governs how much volume we can add to F t by “fattening” to include all points distance at most ε from F t .
■
Warmup with Linear Classification
In this section, we begin to apply our results from Section 3 to get tight regret bounds with computationally efficient algorithms for learning halfspaces in the realizable, smoothed online setting:
Theorem 5.
Let µ be the uniform measure on B 1 .
Suppose that we are in the smoothed, realizable online learning setting, where the adversary samples x t from a distribution that is σ -smooth with respect to µ .
If we predict ˆ y t according to Algorithm 1, then for all horizons T , with probability at least 1 − δ ,
Reg T ≤ 136 d log( d ) + 34 log (cid:18) T σδ (cid:19) + 56 .
Computational Efficiency.
The subroutine JohnEllpsoidCenter ( W t +1 ) can be run in time polynomial in T and d by solving a Semi-definite Program (SDP)
[Boyd and Vandenberghe, 2004, Primak and Kheyfets, 1995].
Note that we change our predictor f t only at the times t that we make a mistake; thus, the number of calls to the SDP is also logarithmic in T .
Proof Sketch of Theorem 5.
We apply Lemma 3 with z t = ( x t , y t ) and ℓ t ( z ) =
I [ˆ y t ̸ =
y t ] .
In order to do this we need to show that ℓ t satisfies ( R, c ) geometric decay, which amounts to finding a geometrically decreasing sequence of upper bounds on µ ( D t ) .
By Lemma 4, it will suffice to provide such bounds on both µ ( F t ) and vol d − 1 ( ∂ F t ) , which is where the specific choice of w t becomes important.
It is now classical [Tarasov et al., 1988, Khachiyan, 1990] that if a polytope is cut by a hyperplane through the center of its John ellipsoid then both halfs have John ellipsoids whose volumes are at most 89 times the volume of the original ellipsoid; as we know that F t ⊂ d · E t [John, 1948], where E t is the John ellipsoid of F t , we see that µ ( d · E t ) is an upper bound on µ ( F t ) that decreases by 89 every time we make a mistake.
The true utility of the center of the John ellipsoid is that it also allows us to show that ∂ F t decreases by a constant factor.
Indeed, we show that vol d − 1 ( ∂ F t ) ≤ vol d − 1 ( ∂ E t ) using a simple projection argument; we then apply a result of Rivin [2007] to bound the size of ∂ E t by µ ( E t ) .
The details are in Appendix D.
■
Importance of the John’s Ellipsoid.
We show in Appendix D.3 that arbitrary predictions y t =
˜ f t ( x t ) , for ˜ f t ∈ F t in the version space, can guarantee 1 /σ -regret at best.
Hence, selecting the correct w t is key.
One natural choice of w t is the Chebyshev center of F t [Elzinga and Moore, 1975], equivalent to a max-margin estimator; unfortunately it need not decrease the volume sufficiently if F t is too ‘pointy.’
Another choice, the centroid of F t , ensures decrease of the polytope ’s volume, but is #P-hard to compute [Rademacher, 2007], and does not ensure decay of the surface area.
The former problem can be accomodated with a sampling scheme [Bertsimas and Vempala, 2004], but the latter is critical.
In contrast, the center of the John ellipsoid controls the decay of both F t and its boundary.
To gain intuition as to why the decay in surface area of F t is necessary, consider the case where F t is simply an arc in S d − 1 .
In this case, as D t is the set of points orthogonal to at least one point in in F t , it follows that D t has positive measure even though F t , being a lower dimensional set, does not; thus, it is impossible in general to get a guarantee on the size of D t only in terms of the volume of F t , without regard to the surface area.
In this way, we see that the choice of w t as the center of the John’s
Ellipsoid is critical to the success of our algorithm.
Remark 3 ( Disagreement Coefficient ) .
Our analysis is similar in spirit to the disagreement-coeffcient analysis of active learning [Hanneke, 2007], which also exhibits geometric decay of the disagreement region D t .
The key difference is that the latter applies to any algorithm that selects a classifier from the version space F t at each time t .
Again, as shown in Appendix D.3, no such analysis can recover a better than 1 /σ -regret bound.
The culprit is that disagreement-coefficient arguments ensure that
D t shrink only probabilistically under samples x t ∼ µ , and this probability may shrink by a factor of σ in the smoothed-online setting.
In contrast, our choice of classifier as the center of the John’s ellipsoid ensures a deterministic decay of the disagreement region whenever a mistake is made.
Lower Bound.
Before we move on to the more complicated settings, we note that this regret bound is tight up to a logarithmic factor in d .
A proof of the following proposition, based on Ville’s inequality, can be found in Appendix D.
Proposition 6.
Suppose that we are in the situation of Theorem 5.
Then there is a realizable adversary such that any classifier experiences
[Equation]
4.1 Smoothed classification via the Perceptron algorithm
Next, we present a guarantee for the classical Perceptron algorithm Rosenblatt [1958], which requires a much weaker notion of smoothness.
We say that the adversary satisfies σ dir directional smoothness if, for any fixed w ∈ S d − 1 , it holds that for all t , ⟨ x t , w ⟩ is σ dir -smooth with respect to the Lebesgue measure on the real line.
As we explain in Example 2 in Appendix G, the directional smoothness σ dir can be nontrivial even when the smoothness parameter σ = 0 .
We now show that the perceptron satisfies the following mistake bound under directional smoothness.
Theorem 7.
Fix any w ⋆ ∈ S
d − 1 and b ⋆ ∈ R .
And suppose that the adversary satisfies σ dirdirectional smoothness.
Then, with probability 1 − δ , the online Perceptron (Algorithm 9 in Appendix G) satisfies
[Equation]
where N err ( w ⋆ , b ⋆ ) =
1 + (cid:80)
Ti =1 I { y t ̸ = sign( b ⋆ + ⟨ w ⋆ , x t ⟩ ) } controls deviation from realizability.
For simplicity, Theorem 7 is stated relative to a fixed w ⋆ ∈ S
d − 1 and b ⋆ ∈ R ; uniform bounds can be derived via a covering argument, at the expense of an additive d log( T/δσ dir ) term in the error bound.
Unlike other algorithms proposed in this paper, Theorem 7 accomodates possibly non-realizable adversaries.
It is also slightly more computationally expedient, not requiring the computation of the center of a John’s ellipsoid.
In contrast, its bound is polynomial in T and 1 /σ dir , rather than logarithmic in T and 1 /σ .
There are situations where Algorithm 1 performs exponentially better than the Perceptron approach: suppose x t is uniform on an ε -ball whose center is chosen by the adversary.
Then we have σ = ε − d and so Theorem 5 implies that the John ellipsoid approach gives regret that scales as O ( d log( d/ε ) + log( T )) , whereas σ dir ≈ 1 /ε and so Theorem 7 only ensures regret that is polynomial in ε .
For further comparison, consult Remark 4 in Appendix G.
Beyond the Linear Case
While the results of Section 4 are technically interesting and have broad applications, they are limited to the specific linear setting.
In this section, we show how our results can be extended, first to the more general affine setting, where the decision boundaries do not have to go through the origin, and then to a more general regime where we do not require linear decision boundaries.
5.1 Affine Classification
Our first generalization of Theorem 5 is to the setting where we allow our decision boundaries to be offset.
Thus instead of assuming realizability with respect to F d lin , we will assume that the adversary isrealizable with respect to
[Equation]
We have the following result:
Corollary 8.
Let µ be the uniform measure on B d 1 and suppose that we are in the smoothed online learning setting, where the adversary samples x t from a distribution that is σ -smooth with respect to µ .
Suppose that the adversary is realizable with respect to the function class F d aff defined in (5.1) .
Then Algorithm 3 in Appendix E.1 is a computationally efficient algorithm for choosing f t ∈ F d aff such that for all T , with probability at least 1 − δ , it holds that
[Equation]
As F d lin ⊂ F d aff , the lower bound of Proposition 6 holds and Corollary 8 is tight up to a factor logarithmic in dimension.
The proof is given in Appendix E.1 and proceeds by reducing to the linear setting of Theorem 5 by imbedding the problem into an online learning problem with contexts ˜ x t ∈ R d +1 , carefully randomized so as to preserve their smoothness with respect to µ d +1 .
5.2 Linear Classification Under a Feature map
One limitation of the above discussion has been the assumption of linearity, which can be overly strong in many cases.
In this section, we weaken this assumption in two ways.
First, we show that if we transform the features with a well-behaved function, then we may still apply our above machinery.
Second, we will show that our approach actually generalizes to polynomial decision boundaries through an elegant reduction.
In both cases, the key technical challenge is to show that our transformed features remain smooth with respect to the uniform measure on a ball.
Note that it is immediate that ϕ ( x t ) is smooth with respect to ϕ ⋆ µ d ; in order to apply our results, however, we require smoothness with respect to the uniform measure.
As it is not true that ϕ ⋆ µ d is smooth with respect to µ d for general ϕ , we require additional assumptions.
We have the following result:
Theorem 9.
Let ϕ : B d 1 → B d 1 be a function such that each coordinate function, ϕ i : R → R satisfies ϕ ′
i ( u ) ≥ α for some α > 0 .
If we run Algorithm 4 in Appendix E.2 then, for all T , with probability at least 1 − δ , it holds that
[Equation]
Algorithm 4, the algorithm that achieves the above regret bound, is actually quite simple as it just runs Algorithm 1 on the data sequence ( ϕ ( x t ) , y t ) .
A proof of a more general result, which applies to a larger class of maps ϕ , is available in Appendix E.2.
Even in the setting of Theorem 9, though, standard transformations like the sigmoid already apply.
We now turn to the more challenging case of polynomial features.
We have the following result:
Theorem 10.
Let ϕ : B d 1 → B m 1 be an L -Lipschitz function whose coordinates are polynomials of degree at most ℓ in the coordinates of x ∈ B d 1 .
Suppose that we are in the smoothed online learning setting where the x t are σ -smooth with respect to µ d and the y t are realizable with respect to F m lin ◦ ϕ .
Suppose further that the Jacobian of ϕ satisfies for some α > 0 ,
[Equation]
Then Algorithm 5 in Appendix E.3 is a computationally efficient algorithm such that for all T , with probability at least 1 − δ ,
[Equation]
Algorithm 5 is a bit more complicated than simply applying Algorithm 1 to ( ϕ ( x t ) , y t ) because if d ≤ m , then ϕ ( x t ) can never be smooth with respect to µ m by dimension constraints.
To escape this difficulty, we define a “meta-point,” x τ , which is the average of ϕ ( x t ) for multiple different t .
To understand why this might fix the problem, consider the identity imbedding of S d − 1 ⊂ B d 1 : if we sample x uniformly on S d − 1 , then the law of x will not even be absolutely continuous with respect to µ d
but if we sample two points x, x ′ ∼ S
d − 1 then their average is absolutely continuous with respect to µ d .
We note that the conditions on ϕ are fairly mild due to the logarithmic dependence on both the Lipschitz constant and the lower bound on the determinant, which is typically no less than exponentially small in d and m .
As a key step on the way to proving Theorem 10, we develop a novel anticoncentration bound for the determinants of polynomial matrices which may be of independent interest: Proposition 11 (Informal Version of Proposition 42) . Suppose that Ψ : R d → S D + is a function whose image is contained in the set of D × D positive, semi-definite matrices and whose entries are polynomials of degree at most ℓ in the coordinates of the argument. Suppose that x 1 , . . .
, x p is a sequence of random variables such that x t is σ -smooth with respect to a common log-concave measure µ , conditional on the history.
If p = Ω (cid:16) Dℓ log (cid:16)
Bℓ
[Equation]
E
[Equation]
x
[Equation]
and || Ψ( x ) ||
op ≤ B almost surely, then with probability at least 1 − δ it holds that
[Equation]
In words, Proposition 11 provides a small-ball type anticoncentration bound for the determinant of a polynomial PSD matrix under a smoothness assumption on the sequence of data.
We now provide a brief sketch of the proof of Theorem 10:
Proof Sketch of Theorem 10.
Algorithm 5 proceeds initially in a similar way to Algorithm 1: we maintain a version space F t ⊂ F m lin that gets updated every round and, when we change w t , we set it to be the center of the John ellipsoid of the version space.
In contradistinction to the earlier algorithm, however, we do not update w t every time we make a mistake.
Instead, for some parameter p , we wait until we have misclassified a label p times, i.e., we guessed − 1 but y t was 1 p times (or the reverse) and construct x τ to be the average of the ϕ ( x t ) for each of these p mistakes.
Using a novel anti-concentration bound for determinants of certain random matrices (Proposition 42) as well as some techniques from geometric measure theory (Proposition 41), we show that x τ is smooth with respect to µ m .
We then apply the abstract decay lemma (Lemma 3) in much the same way as we did in the proof of Theorem 5.
The details are in Appendix E.3.
■
Beyond Binary Classification
In the previous sections, we restricted our focus to binary classification; in this section we expand our scope to a K -class setting and then further extend to a regression setting.
Our results for the regression setting, combined with the reduction of Foster and Rakhlin [2020], are applied to the setting of contextual bandits in Appendix A.
6.1 Multi-Class Classification
We first generalize our results to multi-class classification.
The targets are y t ∈
[ K ] some fixed K and classifications are assigned by maximum inner-product:
[Equation]
Our algorithm is a direct reduction to binary classification.
For each i < j , we maintain an instance
A ( i,j ) bin of Algorithm 1 which makes binary predictions ˆ y
( i,j ) t of y ( i,j ) t = sign( ⟨ w i⋆ − w j⋆ , x t ⟩ ) .
We then set our K -class prediction ˆ y t as the first index i for which
ˆ y ( i,j ) t = 1 for all j > i .
The key insight is that, even though the learner does not recieve feedback on all ˆ y ( i,j ) t in this way, we can always assign a mistake ˆ y t =
y t to an error y
( i,j ) t ̸
= ˆ y
( i,j ) t for some
i < j .
Formal pseudocode is given Algorithm 6 and a proof of the following regret bound is given Appendix F.1.
Theorem 12.
Suppose we are in the realizable, smoothed, online learning setting where the adversary is realizable with respect to the F dK - lin in (6.1) .
Then, then for all T , with probability at least 1 − δ , the regret of Algorithm 6 is at most
[Equation]
The efficiency of the above algorithm follows from the efficiency of the binary classifers
A ( i,j ) bin .
We conjecture that the dependence on K 2 is an artifact of our reduction to (cid:0) K 2 (cid:1) base classifiers.
6.2 Piecewise Regression
This section extends K -class classification to piecewise affine regression.
We now suppose that the targets y t are real-valued, and realizable with respect to the following class of functions:
[Equation]
In contradistinction to the rest of the paper, where the adversary is allowed to play the y t adaptively subject only to the condition of realizability, in this section we suppose that the adversary is semioblivious in the sense that there is a ground-truth function chosen before the start of play and after learning begins, the adversary is only allowed to choose the contexts, x t .
This assumption is natural in the aforementioned contextual bandits application in Appendix A.
Theorem 13.
Adopt the semi-oblivious, smoothed online learning setting, where the adversary begins by choosing g ⋆f ⋆ ∈ G F from (6.3) , and, at each time t , draws x t from a distribution that is σ -smooth with respect to µ and sets y t =
g ⋆ f ⋆ ( x t ) .
Then, Algorithm 7 is an algorithm that is efficient in the number of calls to an ERM oracle over G F that satisfies for all T , with probability at least 1 − δ ,
[Equation]
In Appendix F.2 we prove a more general version of the above result that allows the regression functions on each piece to be polynomial.
The intuition is to reduce K -piece regression to K -class classification, but where each of the “classes” materialize sequentially, once there are sufficiently many points observed to “determine” one of the pieces.
The algorithm and proof are considerably more subtle, and are given in Appendices F.3 and F.4, respectively.
We note that Algorithm 7 only requires the ERM oracle to be called on sets of size independent of T , making the total runtime of the algorithm logarithmic in the horizon.
Acknowledgements
AB acknowledges support from the National Science Foundation Graduate Research Fellowship under Grant No. 1122374.
MS is supported by Amazon.com Services LLC, PO# #D-06310236 and the MIT Quest for Intelligence.
We also would like to thank Sofiia Dubova for her help in translating
Tarasov et al.
[1988] and Michel Goemans for pointing to the closely related, English-language work of Khachiyan [1990].
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] Our claimed contributions are efficiently achievable regret bounds for certain function classes in the smoothed online learning setting.
Right before the related work section, we have a paragraph describing the structure and pointing to where each of the claims is discussed.
(b) Did you describe the limitations of your work?
[Yes] Note that we are only claiming results for particular function classes in the smoothed online learning setting (see contributions).
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See the statements of each proposition and lemma, along with the relevant preliminaries as needed.
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are included in the appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Contents 1 Introduction 2 Preliminaries
The Technical Workhorses 3.
An Abstract Decay Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Volumetric Lemma . .
4 Warmup with Linear Classification 4.
Smoothed classification via the Perceptron algorithm . . . . . . . . . . . . . . . .
5 Beyond the Linear Case 5.
Affine Classification . .
Linear Classification Under a Feature map . . . . . . . . . . . . . . . . . . . . . .
6 Beyond Binary Classification 6.
Multi-Class Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Piecewise Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Contextual Bandits
A.1 Proof of Corollary 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Preliminaries
B.1 Probability and Concentration . .
B.2 Geometric Measure Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Convex Geometry . .
C Technical Workhorses
C.1 Proof of Lemma 3: The Abstract Decay Lemma . . . . . . . . . . . . . . . . . . .
C.2 Proof of Lemma 4: The Key Geometric Lemma . . . . . . . . . . . . . . . . . . .
D Proofs From Section
D.1 Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Theorem 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.3 Lower bound against naive play. . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Proofs from Section
E.1 Proof of Corollary 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Proof of Theorem 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 Proof of Theorem 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3.
Proof of Propossition 41 . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3.
Proof of Proposition 42 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Proof of Theorem 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Formal Guarantees for Piecewise Regression . . . . . . . . . . . . . . . . . . . . .
F.
Algorithm for Piecewise Regression . .
F.
Proof of Proposition 50 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.4.
Guarantee for ERM procedure . . . . . . . . . . . . . . . . . . . . . . . .
F.4.
Distinctness of clustering . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.4.
Key summary of Algorithm 7 . . . . . . . . . . . . . . . . . . . . . . . .
F.4.
Proof of Proposition 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G Non-realizable mistake bounds for the Perceptron.
G.1 Proofs for the Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.2 Lower bound on 1 / ∥ w ⋆ ∥ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Contextual Bandits
In this section, we apply Theorem 13 and the approach of Foster and Rakhlin [2020] to the setting of contextual bandits with contexts drawn from a smooth distribution, considered in Block et al.
Unlike in that work, however, we will realize regret bounds achievable by an oracle-efficient algorithm that are polynomially improved both in the horizon and the number of actions in the particular case of noiseless rewards that are piecewise linear.
We consider the following setting: the learner has access to the context set B d 1 and an action set A with | A | =
A < ∞ .
Let
[Equation]
where G F is as in Theorem 13, be a class of functions
g f : B d 1 ×
A → R .
Before the game begins,
Nature selects some ℓ ⋆ ∈ G AF unknown to the learner.
At each time t , Nature draws x t from a σ -smooth distribution on B d 1 ; the learner then chooses a t ∈ A , observes ℓ ⋆ ( x t , a t ) and suffers the same loss.
Given ℓ ⋆ ∈ G AF , it is clear that the best policy, given a context is greedy:
[Equation]
The goal of the learner is to minimize regret, Reg T , to the optimal policy π ℓ ⋆ .
The primary difference between our setting and that of Foster and Rakhlin [2020], Block et al.
[2022], other than the fact that we are considering a particular function class G AF , is that our losses are noiseless , while the prior works allow for some noise that is mean zero conditional on the history.
We have the following regret bound:
Corollary 14.
Suppose that we are in the contextual bandit setting outlined above with G F from (6.3) and X × A identified with some subset of B d 1 .
Then there is an oracle-efficient algorithm that, for all
T , with probability at least 1 − δ , achieves
[Equation]
We prove Corollary 14 in Appendix A.1 using the reduction of Foster and Rakhlin [2020, Theorem 1] and Theorem 13.
Note that, in contradistinction to the corresponding bound proved as Block et al.
[2022, Theorem 12], we achieve the optimal √ T regret, albeit with stronger assumptions on the setting.
A.1 Proof of Corollary
In this section, we prove Corollary 14 by applying the black box reduction of Foster and Rakhlin [2020] to our Theorem 13.
The key lemma is as follows:
Lemma 15.
Suppose that we are in the setting of Corollary 14 and that we predict ˆ y t ( a ) and sample a t according to Algorithm 2.
Then, for all T , with probability at least 1 − δ , we have
[Equation]
Proof.
We begin by noting that
T
[Equation]
let U = (cid:26) for all 1 ≤ t ≤ T and a ∈ A if p t,a ≤ δ 2 AT then a t ̸ = a (cid:27) A union bound implies that P ( U ) ≥ 1 − δ 2 .
Restricting to U , we note that for any B ⊂ B d 1 measurable,
[Equation]
p
[Equation]
B
[Equation]
σδ thus after restricting to U , the distribution of x t conditioned on a t = a is (cid:0) δσ 2 AT (cid:1) -smooth with respect to µ d .
Thus for each a , we may apply the regret bound from Theorem 13 and, summing over a ∈ A concludes the proof.
■
1: Init: A instances of the Piecewise Regressor (Algorithm 7) regressor ( a ) for a ∈ A , learning rate γ > 0 , exploration parameter µ > 0 .
2: for each time t = 1 , 2 , . . .
do 3: recieve x t 4: for each action
a ∈
A do 5: predict ˆ y t ( a ) =
regressor ( a ) .
predict ( x t ) % Prediction step of Algorithm6:
Assign b t ← argmin a ∈
A ˆ y t ( a ) 7: for each a ̸ = b t do 8:
Assign p t,a ← µ + γ (ˆ y t ( a ) − ˆ y t ( b t ))
( % Inverse Gap Weighting )
Assign p t,b t ← 1 − (cid:88) a ̸ = b t p t,a ( % Inverse Gap Weighting ) 9: sample a t ∼ p t and play a t 10: observe ℓ t ( a t ) 11: update regressor ( a ) .
update ( x t , a t , ℓ t ( a t ))
% Update step of Algorithm
We can now prove Corollary 14:
Proof of Corollary 14.
Note that by Lipschitzness and boundedness, twice the mistake bound is larger than the square loss regret considered in Foster and Rakhlin [2020].
Applying Foster and
Rakhlin [2020, Theorem 1] concludes the proof.
■
B Preliminaries
In this section, we provide some key definitions and results that come up in our analysis.
We divide the section by theme, with the first part collection results on probability and concentration, the second part on geometric measure theory, and the third on convex geometry.
B.1 Probability and Concentration
We begin by stating the foundation of our regret bounds.
Lemma 16 (Ville’s Inequality [Ville, 1939]) .
Let F t denote a filtration and suppose that the sequence of random variables A t is a supermartingale with respect to F t .
Suppose that
[Equation]
Then for any x > 0 , the following inequality holds:
[Equation]
We will also require a standard Chernoff bound.
Lemma 17 (Chernoff Bound) .
Let X 1 , . . .
, X t be a sequence of binary random variables such that E
[ X i | X 1 , . . .
, X i − 1 ] ≥ η .
Then,
[Equation]
Finally, we will clear up any confusion about which distribution is smooth: that of contexts x t or that of samples ( x t , y t ) .
Lemma 18.
Suppose that x ∼ p and ( x, y ) ∼ (cid:101) p where p, (cid:101) p are distributions.
Suppose that y ∈ {± 1 } .
Then if p is σ -smooth with respect to µ then (cid:101) p is (cid:0) σ 2 (cid:1) -smooth with respect to µ ⊗ Unif( {± 1 } ) .
Conversely, if (cid:101) p is σ -smooth with respect to µ ⊗ Unif( {± 1 } ) then p is σ -smooth with respect to µ .
Proof.
The converse follows immediately from Lemma 36, proved in Appendix E.1.
To prove the first statement, note that any distribution on {± 1 } is 12 -smooth with respect to Unif( {± 1 } ) .
Thus, decomposing (cid:101) p ( x, y ) = p ( x ) · (cid:101) p ( y | x ) concludes the proof.
■
B.2 Geometric Measure Theory
The key definition is that of Hausdorff measure, which formally generalizes our intuitive notion of volume and surface area.
Definition 19 (Hausdorff Measure [Federer, 2014]) .
Let X be a metric space.
For any k ∈ R + , we define the k -dimensional Hausdorff measure of a set A ⊂ X to be
[Equation]
where
[Equation]
and diam U i is the diameter of the set
U i , i.e., the maximal distance between any two points contained in U i .
We define the Hausdorff dimension dim( A ) = inf { k > 0 | vol k ( A ) > 0 } .
As is common, when we integrate with respect to the Hausdorff measure, we denote the measure in the integral as d H k instead of d vol k .
Note that when X = R d then vol d exactly coincides with the Lebesgue measure [Federer, 2014].
The following is an immediate consequence of the definition:
Lemma 20.
For a given set A ⊂ X , let N ( A, ε ) denote the minimal number of balls of radius ε required to cover A .
Then
[Equation]
Proof.
It is immediate from the definition that vol εk is monotone nonincreasing as ε ↓ 0 .
The result follows by letting U i be the set of balls of radius ε covering A .
■
We also use the co-area formula:
Theorem 21 (Co-area Formula [Federer, 2014]) .
Let ϕ : R n → R m be a Lipschitz function with n ≥ m .
Then, for A ⊂ R m ,
[Equation]
This in turn implies the projection formula:
Corollary 22 ([Federer, 2014]) .
Let ϕ : X → Y denote a 1 -Lipchitz map between m -dimensional sets X , Y .
Then vol m ( ϕ ( X ))
≤ vol m ( X ) .
Proof.
By Theorem 21,
[Equation]
where the last inequality holds because the Lipschitz assumption bounds the largest singular value of
Dϕ and for any y ∈ ϕ ( X ) , there is at least one point x ∈ ϕ − 1 ( y
[Equation]
■
B.3 Convex Geometry
We first define a polytope:
Definition 23.
We say that a set A ⊂ R d is a polytope if it is the intersection of a finite number of halfspaces.
If A is the intersection of K halfspaces, we say that it has K faces.
We now define an ellipsoid:
Definition 24.
Let A ∈ R d × d be a positive definite matrix and let a ∈ R d be a point.
We define an ellipsoid to be
[Equation]
Note that the volume of an ellipsoid is given by vol d ( E ( A, a ))
=
ω d (cid:112) det( A ) .
We now define the John ellipsoid associated with a convex body:
Theorem 25 (John Ellipsoid [John, 1948, Ball et al., 1997]) .
Let A ⊂ R d be a convex body, i.e., a convex set with nonempty interior.
Then there is a unique ellispoid E A that has maximal volume subject to the condition that the ellipsoid is contained in A .
Furthermore, A ⊂ d · E A .
We require the following general fact about ellipsoids:
Lemma 26 (Corollary 15 from Rivin [2007]) .
Suppose E = E
( A, a ) is an ellipsoid and A has eigenvalues given by q = ( q 1 , . . .
, q d ) .
Then,
[Equation]
Finally, we have the following result about cutting planes through the center of the John ellipsoid:
Lemma 27 (Tarasov et al.
[1988], Khachiyan [1990]) .
Let A ⊂ R d be a polytope with John ellipsoid E
A with center a .
Let A ′ be the intersection of A and a halfspace going through a , i.e., there is some w ∈ R d such that
[Equation]
If E A ′ is the John ellipsoid of A ′ , then
[Equation]
C Technical Workhorses
This appendix proves the technical workhorses, the abstract decay lemma (Lemma 3), and the main geometric lemma, Lemma 4.
C.1 Proof of Lemma 3: The Abstract Decay Lemma
We prove a slightly more general form of the lemma, with a weaker assumption on the sequence of z :
Lemma 28.
Suppose that a sequence ( ℓ t , z t ) satisfies ( R, c ) -geometric decay with respect to some a µ on Z , and define a sequence of stopping times t m where t m = t if t is the m th time that ℓ s ( z s )
= 1 .
Let m t denote the maximal m such that t m < t and thus t m t is last time before t that ℓ s = 1 .
Suppose that for all t , the distribution of z t conditional on t m t is σ -smooth with respect to µ .
Then for all
T ∈ N , with probability at least 1 − δ ,
[Equation]
Proof.
Fix a sequence of positive integers h k for k ∈ N , whose values we tune at the end of the proof.
Let τ 0 = 0 and for all m > 0 , let
[Equation]
Furthermore, let T ( m
[Equation] k − 1 ) h k and t
[Equation]
In words, we consider epochs of length h m , whose length can change every time we make a mistake in an epoch.
We have T ( m ) the time of the m th change of epoch and τ m the number of epochs of length h m we have to go before we make a mistake; we also have t m is the time of the first mistake after the m th change of epoch size.
Let
A m =
[Equation]
be the number of mistakes in a given epoch other than the first mistake.
Let π m = min (cid:0) R m σ , 1 (cid:1) , where we abbreviate R m =
R t m .
We first claim that with probability at least 1 − δ , for all m it holds that:
[Equation]
To see this, let
[Equation]
We show that B λm is a supermartingale for all λ > 0 .
To see this, we have
[Equation]
where the inequality follows because the conditional probability of a mistake for t m +1 ≤ T ( m ) − 1 ≤ π m by the assumption of smoothness conditional on a sub-sigma algebra of that generated by t m −and realizability and T ( m ) − 1 − ( t m + 1) ≤ h m − 1 by construction.
Thus we may apply Ville’s inequality from Lemma 16 and recover (C.4).
Claim 1.
With probability at least 1 − δ , it holds for all m that
[Equation]
Proof of Claim 1.
For any τ m − 1 + ( k − 1) h m ≤ t < τ m − 1 + kh m , smoothness implies
P
[Equation]
where we note that the event that ℓ s ( z s ) = 0 for s < t is contained in the sigma-algebra generated by t m t .
A union bound then implies that
[Equation]
Hence, letting X m be a random variable distributed geometrically with parameter (cid:101)
π m = min( h m π m , 1) , τ m − τ m − 1 stochastically dominates X m .
Thus, for any λ < − log(1 − π m ) ,
[Equation]
We further note that
[Equation]
Thus, setting λ = − 1 , we see that with probability at least δT ,
[Equation]
Because τ m − τ m − 1 > 0 by construction, we may then take a union bound to conclude the proof of the claim.
■
Now we note that
[Equation]
and, further, that if m T is the maximal m such that the preceding display holds,
[Equation]
Thus, combining (C.4) and (C.5), along with the fact that π k ≤ c k R 0 /σ , with probability at least 1 − 2 δ , we have
[Equation]
Now, let h k
= 1 for k ≤ 2 log (cid:0) TR 0 σδ (cid:1) / log (cid:0) 1 c (cid:1) and let h k =
c − k 2 otherwise.
Then we see that if (C.13) and (C.14) hold, then
[Equation]
15) and
[Equation] [Equation]
Thus we see that with probability at least 1 − δ .
[Equation]
which proves the result.
■
We note that Lemma 3 follows immediately because t m t < t almost surely and so the sigma algebra generated by t m t is contained in that generated by the history up to t − 1 and so the smoothness assumption of Lemma 3 implies that of Lemma 28.
C.2 Proof of Lemma 4: The Key Geometric Lemma
We begin by proving the following technical geometric lemma, which, for simplicity, considers subsets of the sphere, rather than of the ball.
This ultimately suffices due to positive homogeneity of linear classifiers.
Lemma 29.
Let ˆ F ⊂ S d − 1 be a measurable subset of the ( d − 1) -dimensional sphere imbedded in
R d .
Let D (ˆ F ) denote the set of points in S d − 1 orthogonal to at least one point in ˆ F , i.e.,
[Equation]
Then, if vol k is the k -dimensional Hausdorff measure on the sphere, we have
[Equation]
Proof.
For a given set A ⊂ S d − 1 , denote by T ( A, ε ) the “tube” of radius ε around A , i.e., the set of points in S d − 1 with distance at most ε from a point in A .
Note that for any fixed point w ∈ ˆ F , we have D ( w ) is just the ( d − 2) -sphere formed by intersection the linear space orthogonal to w with S d − 1 .
If ˆ B ε ( w ) denotes the ε -ball around w in S d − 1 , then we claim that D ( ˆ B ε ( w ))
⊂
T ( D ( w ) , ε ) .
Indeed, suppose that v ∈ ˆ B ε ( w ) so that ⟨ w ′ , v ⟩ = 0 for some w [-(cid:
 )-]′ ∈ ˆ B ε ( w ) .
Let α be a member of the orthogonal group such that αw ′ = v and ⟨ αw, w ⟩ = 0 .
Then ⟨ v + α ( w − w ′ ) , w ⟩ =
and || α ( w − w ′ )
||
=
|| w − w ′ || ≤ ε , proving the claim.
Let N ( (cid:98) F , ε ) denote the minimum size of an ε -net of F and let P ( (cid:98) F , ε ) denote the maximum size of an ε -packing.
By abuse of notation, we will also use N ( (cid:98) F , ε ) to denote the minimal ε -net itself.
The fact that D ( (cid:98) B ε ( w ))
⊂
T ( D ( w ) , ε ) implies that
[Equation]
By packing, covering duality, we have
[Equation]
Now, we may apply Gray [2003, Theorem 10.20], the generalization of Steiner’s formula to submanifolds of a sphere, to get
[Equation]
Putting this together, we have
[Equation]
Now we may apply Weyl’s tube formula [Weyl, 1939]
(see Gray [2003], Lotz [2015] for a clear exposition on the topic) to S d − 2 imbedded as the equator of S d − 1 to get that for any ε < 1 ,
[Equation]
As ε ↑ 1 , the above expression tends to 2 d .
Putting everything together, we have
[Equation]
as desired.
■
We now use the homogenity of the inner product to show that it suffices to consider the sphere:
Lemma 30.
Let F ⊂ B d 1 and let (cid:98) F t denote its projection to S d − 1 .
Suppose that F is such that
[Equation]
Then,
[Equation]
Proof.
Let (cid:98) F r = r (cid:98) F .
Then we see from Theorem 21 that vol d − 1 ( (cid:98) F r ) =
r d − 1 vol d − 1 ( (cid:98) F ) .
Thus,
[Equation]
In particular, this holds for F = B d 1 .
Thus, we have
[Equation]
as desired.
■
We now put everything together:
Proof of Lemma 4.
Let (cid:98) D ( F ) be the set of x ∈ D such that || x
||
= 1 and let (cid:98) F be defined similarly.
By the positive homogeneity of both D ( F ) and F , we have
[Equation]
where vol d − 1 ( · ) denotes the ( d − 1)
-dimensional Hausdorff measure.
Thus, it suffices to compare vol d − 1 ( (cid:98) D ( F )) with vol d − 1 ( (cid:98) F ) , which is the content of Lemma 29.
The result follows.
■
D Proofs From Section
In this appendix, we provide proofs of Theorem 5 and Proposition 6.
D.1 Proof of Theorem
In order to apply Lemma 3 to prove Theorem 5, we need to show that the loss functions satisfy ( R, c ) -geometric decay.
We will show this for R =
d +1 d 2 d and c = 89 using Lemma 4 and some more convex geometry.
We begin by proving the following characterization of the disagreement region, which will in turn allow us to apply Lemma 4:
Lemma 31.
Suppose that we are in the situation of Theorem 5.
Then we have
[Equation]
Proof.
Recall that the version space is defined as
[Equation]
If x ∈ D t , then there are w, w ′ ∈ F t such that sign( ⟨ w ′ , x ⟩ ) ̸ = sign( ⟨ w ′′ , x ⟩ ) .
Consider the continuous function h ( λ ) = ⟨ λw ′′ + (1 − λ ) w ′ , x ⟩ ; by the intermediate value theorem, there is some 0 <
λ ⋆ < 1 and w = λ ⋆ w ′′
+ (1 − λ ⋆ ) w ′ such that ⟨ w, x ⟩ = 0 .
By convexity of F t , then w ∈ F t and thus D t is contained within the set of points orthogonal to at least one point in F t .
■
With Lemma 31 in hand, we will be able to apply Lemma 4 and it will suffice to control µ ( F t ) and vol d − 1 ( F t ) .
The next result bounds these quantities in terms of their analogues in E t :
Lemma 32.
Let F ⊂ R d be a convex body with John ellipsoid E .
Then we have
[Equation]
Proof.
Note that it is a classical fact that F ⊂ d · E [John, 1948] and thus µ ( F ) ≤ d d µ ( E ) .
We thus only have to prove the second bound.
To do this we first note that
[Equation]
To see that this is the case, consider π : ∂ ( d · E ) →
∂ F be projection onto the convex F .
Then π is a contraction and thus shrinks Hausdorff measure as per Corollary 22.
We now apply Lemmaand note that because our ellipsoids are contained in the ball,
|| q || ≤ 2 · √ d , where q is the vector of semi-axis lengths, i.e., the eigenvalues of the associated positive definite matrix.
Thus we have
[Equation]
and the result follows.
■
We are now finally ready to apply the geometry that we have done so far to prove a slightly more general form of Theorem 5, which we will require to apply some of our reductions below.
Proposition 33.
Suppose we are in the situation of Theorem 5 with the added complication that at any time t , the adversary can choose to censor round t from the learner, so the learner does not observe y t and does not suffer loss at time t .
We further allow x t to be drawn adversarially with the condition that if x t is drawn adversarially, then the adversary always censors time t .
Then the conclusion of Theorem 5 holds.
Proof.
By Lemma 3, it suffices to show that with our choice of w t , the sequence ℓ t =
I [sign( ⟨ w t , x t ⟩ ) ̸
= y t and round t is not censored ] satisfies ( R, c ) geometric decay with respect to µ for come R, c .
In particular, we need to find an
R ≥ µ ( D 1 ) and a c < 1 such that if we make a mistake, then µ ( D t +1 ) ≤ cµ ( D t ) .
Note that if ℓ t =then we must have x t ∈ D t and y t is not censored.
By Lemma 31 it suffices to control the size of the set of points orthogonal to at least one w ∈ F t ; by Lemma 4, it in turn suffices to control µ ( F t ) and vol d − 1 ( F t ) .
Applying Lemma 32 to the preceding logic, we have:
[Equation]
As µ ( E t ) ≤ 1 , we may choose R =
d +1 d 2 d and reduce to showing that every time we make a mistake, µ ( E t +1 ) ≤ cµ ( E t ) for some c .
Now, suppose that we make a mistake at time t , i.e., ⟨ w t , y t x t ⟩ < 0 .
Then, we have
[Equation]
by monotonicity.
Thus F t +1 is a subset of the intersection of F t and a halfspace through the center of E t .
Thus, by Lemma 27, vol( E t +1 ) ≤ 89 vol( E t ) .
Thus, we may choose c = 89 and conclude the proof.
■
We remark that Theorem 5 trivially follows from Proposition 33 by restricting the adversary to never censor a time t .
D.2 Proof of Theorem
We construct separate adversaries which regret E [Reg T ] ≥ Ω( d ) and E
[Reg T ] ≥ Ω (cid:0) log (cid:0) Tσ (cid:1)(cid:1) .
Randomizing between the two with probability one-half gives the lower bound.
We first note that any algorithm must experience E [Reg T ] ≥ d +12 against some adversary; indeed, as a generic set of d +1 points defines a hyperplane, a realizable adversary can choose y t as independent
Rademachers for each 1 ≤ d ≤ d + 1 and the learner will suffer expected regret d +12 .
We now construct an adversary in one dimension that will force E [Reg T ] ≥ Ω (cid:0) log (cid:0)
Tσ (cid:1)(cid:1) ; by projecting onto some fixed direction, the higher dimensional case reduces to this setting.
Thus, suppose that the x t are required to be sampled from a distribution that is σ -smooth with respect to the uniform measure on the unit interval.
At each time t , let D t be the interval between the rightmost x s labelled − 1 and the leftmost x s labelled 1 , let R t be its length and w t its midpoint.
Fix 0 < ε < 1 to be tuned later and let
[Equation]
be the set of points in the disagreement region close to its boundary.
We let the adversary select the distribution that picks uniformly from (cid:101) D t with probability min (cid:16) µ ( (cid:101) D t ) σ , 1 (cid:17) and with remaining probability selects 0 .
If | x t − w t | ≥ R t 2 , then y t is determined by realizability.
Otherwise, let y t be an independent Rademacher random variable.
Let π m = min (cid:0) εR m σ , 1 (cid:1) and t m be the m th time that x t ∈ (cid:101)
D t , we see that t m +1 − t m is geometrically distributed with parameter π m and thus
[Equation]
is a supermartingale for λ <
min k ≤ m
[Equation] m ) .
Note that by construction,
R m +1 ≥ (1 − ε ) R m and thus R m ≥ (1 − ε ) m .
Setting λ
= π m ≤ − log(1 − π m ) and applying
Ville’s Inequality, Lemma 16, we get that with probability at least 1 − δ , for all m ,
[Equation]
We now note that
[Equation]
using monotonicity, the fact that π m ≤ 1 and the following computation:
[Equation]
Now, using the fact that π m ≥ εσ (1 − ε ) m , we have
[Equation]
Setting ε = 1 − e − 1 , we see that there is some constant c > 0 such that with probability at least 1 − δ ,
[Equation]
In particular, there is a universal constant C such that if m = C log
[Equation]
, then with probability at least 1 − δ we have Reg T ≥ m because the probability of a mistake, given that x t ∈ (cid:101) D t is 12 .
The result follows.
D.3 Lower bound against naive play.
In this section, we show that it is necessary to choose the half-spaces w t intelligently in order to attain logarithmic-in- 1 /σ regret.
Consider d = 1 , so µ 1 is the uniform measure on the interval [ − 1 , 1] .
We define F thres := { x (cid:55)→ sign( x − c ) } as the set of (monotone) threshold classifiers.
Given a function class F , we say that a learning strategy is consistent , if at each t ∈ [ T ] , it selects an f t ∈ F t in the version space F t :=
{ f ∈ F : f ( x s ) =
y s , [Equation]
Define the left and right endpoints of the negative and positive regions
[Equation]
For a given η > 0 , we consider the strategy
[Equation]
This is consistent with F thres , since the thresholds are always chosen strictly between ˜ x t − 1 and ¯ x t − 1 .
However, the strategy is very naive, since it defaults to setting the threshold only slightly to the right of ˜ x t − 1 .
As a consequence, we show it suffers Ω(1 /σ ) regret when η is small.
Proposition 34.
Fix η > 0 .
For T ≥ 1 and σ ∈ (1 /T, 1 / 4] , there exists an F thres -realizable, σ smooth adversary such that the strategy in Equation (D.19) suffers expected regret linear in 1 /σ for η small:
[Equation]
Proof.
At each time 1 ≤ t ≤ T 0 :=
⌊ 1 /σ ⌋ , the adversary selects x t = − 1 + 2 σ ( t − 1) + 2 σa t , a t ∼ Unif([0 , 1]) .
For times t ≥ T 0 , the adversary selects x
t ∼ Unif([ − 1 , 1]) .
This adversary is clearly σ smooth, satisfies x t ∈ − 1 + 2 σ [ t − 1 , t ] until T 0 , and then plays arbitrarily.
Moreover, for σ ≤ 1 /T , x t ∈ [ − 1 , 1] for all t .Fixing a ground-truth classifier f ⋆ ( x ) = sign( x − 1) , we see y s =
f ⋆ ( x t ) = −is realizable for all t .
Lastly, we analyze the regret of Equation (D.19); notice that under the above adversary, ¯ x t = 1 , so we are always in the first case
y t
= sign( x t − max 1 ≤ s ≤ t − 1 x s − η ) .
Then, for any t ≤ T 0 ,
[Equation]
Hence,
E [Reg t ] =
T (cid:88) t =
[Equation]
F
[Equation]
1 ]] ≥
[Equation]
(cid:88)
[Equation]
= [Equation]
= [Equation] t =
[Equation]
≥
[Equation]
T 0 :=
⌊ 1 /σ ⌋ .
■
E Proofs from SectionE.1 Proof of Corollary
The key technical result is contained in the following lemma, which says that we can lift a σ -smooth distribution on B d 1 to one on B d + and only lose a factor that is exponential in dimension.
Because our regret guarantees are only logarithmic in σ , this will translate into a factor that is only linear in d by reducing to the setting of Theorem 5.
We have the following result:
Lemma 35.
There exist a probability kernel K : B d 1 → △ ( B d + ) that satisfies the following two properties: first,
P ˜ x ∼ K ( ·| x ) (cid:0) for all ˜ w = ( w, b ) ∈ R d × R , sign( ⟨ w, x ⟩ + b ) = sign( ⟨ ˜ w, ˜ x ⟩ ) (cid:1)
= 1 , and second, if p is σ -smooth with respect to µ d , then K ◦ p is σ ′ -smooth with respect to µ d +1 , where σ ′ =
σ/ 4 d +2 and (cid:101) x ∼ K ◦ p if x ∼ p and (cid:101) x ∼ K ( ·| x ) .
Proof.
For general b , define (cid:101) w := ( w, b ) , let ϕ ( x, z )
=
z ( x, 1) 4 , and let (cid:101)
x
= ϕ ( x t , z t )
z t ∼ Unif(1 , 2) .
(E.1)
Note that whenever x ∈ B d 1 , P [ (cid:101) x t ∈ B d
+ | x t = x ] = 1 .
Moreover, sign( ⟨ w, x t ⟩ + b ) = sign( ⟨ (cid:101) w, ( x t , 1) ⟩ ) = sign( ⟨ (cid:101) w, z t ( x t , 1) ⟩ ) = sign( ⟨ (cid:101) w, (cid:101) x t ⟩ ) .
Since our proposed algorithm is a function only of the version space F t , and not the disagreement region , it follows that we can assume without loss of generality that the learner interacts with the distribution (cid:101) p t induced by drawing x t ∼ p t , z t ∼ Unif[1 , 2] , and (cid:101) x t = ϕ ( x t , z t ) .
To conclude, we must argue that if x t ∼ p t is σ -smooth with respect to the uniform measure µ d on B d 1 , then ˜ x t ∼ ˜ p t is σ ′ -smooth with respect to the uniform measure µ d +1 on B d + , for an appropriate σ ′ .
Let (cid:101) µ d +1 denote the density of (cid:101) x
= ϕ ( x, z ) , where z ∈ Unif[1 , 2] when x ∼ µ d .
Then,
[Equation]
To bound the first term, consider ϕ − 1 , the inverse of ϕ from B d 1 / 4 × [ 14 , 1 / 2]
→ B d 1 / 4 × [1 , 2] given by
[Equation]
Then, ˜ p t is the pushforward under ϕ − 1 of the measure p t ⊗ Unif[ 14 , 12 ] , and ˜ µ d +1 the pushforward of µ d ⊗ Unif[ 14 , 12 ] .
Thus, by Lemma 36, we have that (cid:101) p t is σ -smooth with respect to (cid:101) µ d +1 .
Now, we compute d (cid:101) µ d +1 ( (cid:101) x ) d µ d +1 ( (cid:101) x ) .
It suffices to show that for any set H ⊂ B d 1 / 2 × [ 14 , 1 / 2] , we have
[Equation]
for some desirable constant C .
Let J ϕ − 1 denote the Jacobian of the map ϕ − 1 : ( x, u ) (cid:55)→ ( x/z, 4 z ) .
Then, J ϕ − 1 is a triangular matrix with determinant 4(1 /z )
d .
Thus, on H ⊂ B d 1 / 2 × [ 14 , 1 / 2] , its
1: Initialize (cid:101) W 1 =
B d + , (cid:101) w 1 = e 1 ∈ (cid:101) W 1 , 2: for t = 1 , 2 , . . .
do 3:
Recieve x t , draw z t ∼ Unif(1 , 2) , and assign (cid:101) x t ← ϕ
( x t , z t ) =
z t ( x t , 1)4: 5: predict ˆ y t = sign( ⟨ (cid:101) w t , (cid:101) x t ⟩ ) , ( % self .
classify ( x t ) ) 6:
Update (cid:101) W t +1 = (cid:101) W t ∩ { (cid:101) w ∈ B d + | ⟨ (cid:101) w, (cid:101) x t y t ⟩ ≥ 0 } 7: if ˆ y t ̸ =
y t then ( % self .
errorUpdate ( x t ) ) 8: (cid:101) w t +1 ← JohnEllpsoidCenter ( (cid:101) W t +1 ) 9: % returns center of John Ellpsoid of given convex body determinant is at most | det J ϕ ( x, z ) | = 4 d +1 . Hence, ˜ µ d +1 ( H ) = P ( x,z )
[Equation] d 1 × Unif[1 , 2] [ ϕ ( x, z ) ∈ H ] = P ( x,z [Equation]
B
[Equation]
1 × Unif[1 , 2] [( x, z ) ∈ ϕ [Equation] ( H )] = (cid:82)
[Equation]
d z vol d +1 ( B d 1 × Unif[1 , 2]) = (cid:82)
[Equation]
det(
[Equation]
[Equation]
{+d z vol d +1 ( B d 1 × Unif[1 , 2]) ≤ 4 d +1 vol( H ) vol d +
+}
( B d 1 × Unif[1 , 2])
=
µ d +1 ( H ) · 4 d
[Equation]
Combining these computations with (E.2) yields
[Equation]
which concludes the proof.
■
Lemma 36.
Suppose that f : X → Y is a measurable map and suppose that p, µ are measures on X and p is σ -smooth with respect to µ .
Define the pushforward measure on Y by taking f ⋆ µ ( B ) = µ ( f − 1 ( B )) for any measurable B ⊂ Y .
Then f ⋆ p is σ -smooth with respect to f ⋆ µ .
Proof.
Let B ⊂ Y be measurable.
Then
[Equation]
As this holds for any B ⊂ Y , the result follows.
■
We now describe Algorithm 3.
At each time t , we draw z t ∼ Unif(1 , 2) independently and form (cid:101) x t = ϕ ( x t , z t ) , where ϕ is as in (E.1).
We then run the classify and update subroutines of Algorithmat each time step on the new data sequence ( (cid:101) x t , y t ) .
We are now ready to prove Corollary 8:
1: Initialize (cid:101) W 1 =
B d + , (cid:101) w 1 = e 1 ∈ W 1 , ϕ : B d 1 → B d2: for t = 1 , 2 , . . .
do 3:
Recieve x t , predict ˆ y t = sign( ⟨ (cid:101) w t , ϕ ( x t ) ⟩ ) , ( % self .
classify ( x t ) ) 4:
Update (cid:101) W t +1 = (cid:101) W t ∩ { (cid:101) w ∈ B d + | ⟨ (cid:101) w, ϕ ( x t ) y t ⟩ ≥ 0 } 5: if ˆ y t ̸ =
y t then ( % self .
errorUpdate ( x t ) ) 6: (cid:101) w t +1 ← JohnEllpsoidCenter ( (cid:101) W t +1 ) 7: % returns center of John Ellpsoid of given convex body
Proof of Corollary 8.
We use Algorithm 3 to reduce the problem to the situation of Theorem 5.
Indeed, by Lemma 35, the data sequence ( (cid:101) x t , y t ) satisfies the property that (cid:101) x t is (cid:0) 4 − d − 2 σ (cid:1) -smooth with respect to µ d +1 and is realizable by the function class F d +1 lin .
The result then follows immediately from Theorem 5.
■
E.2 Proof of Theorem
We now prove begin generalizing beyond linear function classes with Theorem 9.
The key technical result shows that if ϕ : B d 1 → B d 1 is well-behaved, then ϕ ⋆ µ d is σ -smooth with respect to µ d , which will then allow us to apply Theorem 5.
Lemma 37.
Suppose that p is a measure on B d 1 that is σ -smooth with respect to µ d and suppose that ϕ : B d 1 → B d 1 is a function satisfying the following two properties:
• There is some c > 0 such that | det( Dϕ ( x )) | > c for all x ∈ B d 1 .
• There is some N ∈ N such that for every x ∈ B d 1 , it holds that (cid:12)(cid:12) ϕ − 1 ( x ) (cid:12)(cid:12) ≤ N , where ϕ − 1 ( x ) =
(cid:8)
y ∈ B
d 1 | ϕ ( y ) =
x (cid:9) .
Then, ϕ ⋆ p is (cid:0) cN σ (cid:1) -smooth with respect to µ d .
Proof.
By Lemma 36, we have that ϕ ⋆ p is σ -smooth with respect to ϕ ⋆ µ d .
Thus, as
[Equation]
it suffices to bound the latter factor.
By the area formula [Federer, 2014], we have for any B ⊂ B dthat
[Equation]
Thus we see that for any B ,
[Equation]
and so the result follows.
■
As a corollary, we generalize Theorem 5 to adversaries that are now realizable to a class linear in some new set of features:
Corollary 38.
Let ϕ be a map as in Lemma 37 and suppose that we are in the smoothed online learning setting with an adversary realizable with respecto to F ◦ ϕ = { x (cid:55)→ f ( ϕ ( x )) | f ∈ F } .
If we run Algorithm 4 on the data ( ϕ ( x t ) , y t ) , then for all T , with probability at least 1 − δ , it holds that
[Equation]
Proof.
The statement follows immediately from applying Theorem 5 to the data sequence ( ϕ ( x t ) , y t ) and using Lemma 37 to bound the smoothness.
■
Finally, we prove the simpler result stated in Section 5:
Proof of Theorem 9.
By Corollary 38, it suffices to bound N and c in Lemma 37.
Suppose that ϕ ( x ) =
( ϕ 1 ( x 1 ) , . . . , ϕ d ( x d )) as in the statement of the result.
Then we see that D ϕ ( x ) is diagonal with ϕ ′
i ( x i ) as the i th element of the diagonal and thus
[Equation]
where the final inequality follows from the assumption.
Note that if ϕ ′ i > 0 for all i , then ϕ is strictly increasing coordinate wise and thus we may take N = 1 .
The result follows.
■
E.3 Proof of Theorem
In this section, we show that our techniques extend to polynomial decision boundaries.
Morally, we proceed on similar lines as to the proof of Theorem 9 outlined in the previous section, but there are a number of new technical subtleties that appear in this analysis that were not present before.
The most salient difference between the maps considered above and that which is required for a polynomial decision boundary is that polynomial features require imbedding our problem into a higher dimensional space; while Lemma 36 ensures that the pushforward of the law of each x t is smooth with respect to the pushforward of µ d , our analysis is very specific to the dominating measure being uniform on the ball, which can never happen if we are pushing µ d forward into a higher dimensional space.
In order to resolve this difficulty, we will present a reduction that allows us to combine multiple points into one ‘meta-point,’ whose law will be smooth with respect to the uniform measure on the higher dimensional ball.
We will then be able to reduce to a similar setting as considered in Theorem 5 and deduce a similar regret bound.
We prove the following result:
Proposition 39.
Suppose that ϕ : B d 1 → B m 1 satisfies the following properties:
• ϕ is L -Lipschitz.
• ϕ is polynomial in the sense that each of the coordinates of ϕ is a polynomial in the coordinates of x ∈ B d 1 with degree at most ℓ .
• There is an α > 0 such that the Jacobian D ϕ satisfies:
[Equation]
Suppose further that the x t ∈ B d 1 are generated in a σ -smooth manner and the y t are realizable with respect to F m lin ◦ ϕ .
Then there is a universal constant C such that for all T , if we set
[Equation]
and run Algorithm 5, then with probability at least 1 − δ ,
[Equation]
Proof.
Consider the sequence of stopping times ρ τ , where ρ 0 = 0 and, for τ > 0 , ρ τ = inf
[Equation]
I [ y t = − 1 and ˆ y t = 1] 
 ≥ p   .
for some p to be determined.
Furthermore, let ℓ t
= I [ t ∈
[Equation]
2 p + [Equation] ) N and t − 2 p ≤ ρ τ ≤ t for some τ ]
We begin by claiming that the following inequality holds:
[Equation]
Indeed, we note that the sum is equal to τ T , the maximal τ such that ρ τ ≤ T and the pigeonhole principle tells us that we suffer at most 2 p + 1 mistakes in the interval ρ τ − 1 ≤ t ≤ ρ τ .
There are at most 2 p mistakes in the interval ρ τ T ≤ t ≤ T and so the first inequality holds.
The second inequality follows from noting that ℓ t = 1 implies that t
= (2 p + 1) t ′ for some t ′ .
For each 1 ≤ τ ≤ τ T , we let
[Equation]
Now, fix β, γ > 0 to be set later and let
U 1 = { for all t such that P t (ˆ y t ̸ =
y t ) <
γ, it holds that ˆ y t
=
y t }
U 2 = { for all t such that P t ( y t = y ) < β for some y , it holds that y t ̸ = y } .
We claim that for some p , there is a sequence of x ′ τ ∈ B m 1 such that if
[Equation]
then first, P ( U ) ≥ 1 − 2 Tβ − Tγ − δ 4 and second, the x ′ τ are σ ′ -smooth with respect to µ m .
This claim, and the dependence of σ ′ on the relevant parameters is the subject of Proposition 40 below.
For now, we will take it as given.
Now, recalling the disagreement region and version space notation
D t , F t , as from Appendix D, we note that if ℓ t {+′ = 
+}
and τ is maximal subject to ρ τ ≤ (2 p + 1) t ′ , then we must have x τ ∈ D ρ τ − 1 .
To see this, note that w t = w ρ τ − 1 for ρ τ − 1 < t ≤ ρ τ and thus w ρ τ − 1 is such that (cid:10) w ρ τ − 1 , y s ϕ ( x s ) (cid:11) < 0 for each such s .
By linearity, we have
[Equation]
Realizabilty implies that there is some w such that ⟨ w, y s ϕ ( x s ) ⟩ ≥ 0 for all s , and so linearity implies that, for that w ,
[Equation]
Thus we see that x τ ∈ D ρ τ − 1 .
We now note that for any t ′ ,
[Equation]
where σ ′ is as in Proposition 40.
Applying now Lemmas 4, 31 and 32 in the same way as in the proof of Theorem 5, we place ourselves now into the situation of Lemma 28, the generalized version of the master reduction Lemma 3.
Thus, we have that for all T , with probability at least 1 − δ ,
[Equation]
1: Initialize (cid:101) W 1 =
B m 1 , (cid:101)
w 1 =
e 1 ∈ W 1 , ϕ : B d 1 → B m 1 , M 1 , M − 1 = {} , p ∈ N 2: for t = 1 , 2 , . . .
do 3:
Recieve x t , predict ˆ y t
= sign( ⟨ (cid:101) w t , ϕ ( x t ) ⟩ ) , ( % self .
classify ( x t ) ) 4:
Update (cid:101) W t +1 = (cid:101) W t ∩ { (cid:101) w ∈ B d + | ⟨ (cid:101) w, ϕ ( x t ) y t ⟩ ≥ 0 } 5: if ˆ y t ̸ =
y t then ( % self .
errorUpdate ( x t ) ) 6:
Update M y t ← M y t ∪ { x t } 7: if max ( | M 1 | , | M − 1 | ) =
p then 8:
Update 9: (cid:101) w t +1 ← JohnEllpsoidCenter ( (cid:101) W t +1 ) 10: % returns center of John Ellpsoid of given convex body 11:
Reset M 1 , M − 1 ← {}
If we set p = Cmℓ log (cid:18) LℓT δ (cid:19) then plugging in the penultimate display into (E.5), taking γ = δ 4 T and β = δ 8 T , and plugging in the bounds from Proposition 40 concludes the proof.
■
We note that Theorem 10 follows immediately from Proposition 39.
The key difficulty in the proof of
Proposition 39, that we left until now, is the smoothness of the x τ .
We state this fact, and provide a quantitative bound on the smoothness parameter, in the next proposition:
Proposition 40.
Let ϕ : B d 1 → B m 1 be an L -Lipschitz function whose coordinates are polynomials in the coordinates of x ∈ B d 1 , with degree at most ℓ .
Suppose ϕ is such that
[Equation]
Fix any p ∈ N such that pd ≥ m .
Suppose that ( x 1 , y 1 ) , . . .
, ( x T , y T ) is a data sequence satisfying the following four properties:
• The distribution of x t conditional on the history is σ -smooth with respect to µ d .
• For y ∈ {± 1 } , for any t , P t ( y t = y ) ≥ β .
The y t are realizable with respect to F m lin ◦ ϕ .
• For any t , and any choice of ˆ y t by the learner, P t (ˆ y t ̸ = y t ) ≥ γ .
where P t is the conditional probability of the history up to time t .
Now, consider the set of stopping times ρ τ with ρ 0 = 0 and
[Equation]
 
Let y τ
=
y ρ τ and let x τ = 1 p p (cid:88)
[Equation] where τ 1 , . .
, τ p are the p times ρ τ − 1 < t ≤ ρ τ satifying y t =
y τ and ˆ y t ̸ = y t .
There is a universal constant C such that if
[Equation]
then there is a data sequence a sequence x ′ t ∈ B m 1 satisfying the following three properties.
First, the sequence ( x ′ t , y t ) is realizable with respect to F m lin .
Second, with probability at least 1 − δ , for all t , x t = x ′ t .
Third, the x ′ t is σ ′ -smooth with respect to µ m , where
[Equation]
wisth c a universal constant
Intuitively, we wait until we have misclassified a class p times and then form a ‘meta-point’ ( (cid:101) x τ , (cid:101)
y τ ) that will allow us to reduce to the setting of Theorem 5.
The meta-point will be constructed by averaging samples x t in order to ensure smoothness with respect to µ m .
We will show that Proposition 40 follows from three results that we will prove below.
First, we show that if ϕ is well-behaved, and the x t are σ -smooth with respect to µ d , then the (cid:101) x t are σ ′ -smooth with respect to µ m .
Proposition 41.
Suppose that ϕ : B d 1 → B m 1 is a smooth map between Euclidean balls of dimensions d and m with Jacobian D ϕ .
Consider the function f : ( R d ) p → R m defined as
[Equation]
Suppose that that the following three conditions are satisfied:
• There is some V ⊂ ( B d 1 ) × p and c > 0 such that for µ ⊗ p d -almost every x ∈ V , det(D ψ ( x )D ψ ( x ) T ) ≥ α 2 .
• For some ℓ ≥sup
[Equation]
In particular, this holds if ϕ ( x ) is a polynomial of degree at most ℓ .
• Finally, suppose that the joint distribution of ( x 1 , . . .
, x p ) is σ p -smooth with respect to µ ⊗ p d .
If pd ≥ m , then the law of ψ ( x 1 , . . . , x p ) , conditioned on ( x 1 , . . . , x p ) ∈ V is σ ′ -smooth with respect to µ m , the uniform measure on B m 1 , where
[Equation]
Second, we show that if ϕ is a polynomial, then it is well-behaved in the sense of Proposition 41 with high probability, by proving the more general small-ball type estimate below:
Proposition 42.
There exists a univesal constant C such that the following holds.
Let Ψ : R d → S D + be any function whose image is contained in the set of PSD matrices and whose entries are polynomials of degree at most ℓ , and let x 1 , . . .
, x p be a sequence of random-variables such that, for each t , x t | x 1 , . . .
, x t − 1 is σ -smooth with respect to a common log-concave measure µ , and P x ∼ µ [ λ max (Ψ( x )) ≤ B ] = 1 .
Define [Equation]
Suppose that p ≥ 16 log(1 /δ )
+ D 4 log(24 B )
+ 14 (log det(Λ) + Dℓ log( Cℓ )) .
Then,
[Equation]
Finally, we will show that if the probabilities corresponding to each label are well-controlled, then the laws of x τ k are smooth with respect to µ d :
Proposition 43.
Let ( x τ 1 , y τ 1 ) , . . .
, ( x τ p , y τ p ) be the sequence of points defined in Proposition 40, arising from a sequence of ( x t , yt ) with the x t being σ -smooth conditional on the history and for each t , and y ∈ {± 1 } it holds that P ( y t = y ) ≥ β and that P (ˆ y t ̸ = y t ) ≥ γ .
Then, for each i , it holds that the law of x τ i conditional on the history up to time τ i − 1 is ( βγσ/T ) -smooth with respect to µ d .
In particular, the law of ( x τ 1 , . .
, x τ p ) , conditional on the sigma-algebra generated by ρ τ − 1 , is ( βγσ/T ) p -smooth with respect to µ ⊗ p d .
Proof.
Note that a peeling argument and induction show that the second statement follows immediately from the first.
For any τ, i , denote probability conditioned on the history up to time τ i − 1 by P τ i − 1 .
Let B ⊂ B
d 1 be measurable.
Then we compute that P τ i − 1 ( x τ i ∈ B ) can be given by:
[Equation]
γ
[Equation]
[Equation]
y
[Equation]
[Table]
[Equation]
P
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Table]
y [Table] [Equation] [Table]
( (cid:101)
y
[Table]
[Equation]
[Equation]
Thus, the result follows.
■
Propositions 41 and 42 will be shown below but for now we will take them as given.
We can now prove the key proposition:
Proof of Proposition 40.
Let
[Equation]
ℓm
[Equation]
Cℓ [Equation] ℓm α
[Equation]
for all τ [Equation] and note that by the fact that τ ≤ T p ≥ Cm log
[Equation]
, applying a union bound to Proposition 42 and using Proposition 43 to ensure that the hypothesis holds, shows that P ( U ) ≥ 1 − δ .
On U we will let x ′ τ
= x τ and on U c , we will draw x ′ τ from µ m , conditioned on ( x ′ τ , y τ ) being realizable with respect to F m lin .
Note that we have realizability by construction on U c .
On U , we have that x ′ τ
= x τ and note that convexity implies that if y τ 1 = · · · = y τ p , then any realizable adversary must classify x τ as y τ .
Indeed, if w ∈ F m lin is in the version space, and y τ = 1 , then
[Equation]
and similarly if y τ = − 1 .
Thus, realizability holds.
As we have already seen that { there exists τ such that (cid:101) x ′ τ ̸ = x τ } ⊂ E c and P ( U c ) ≤ δ , it suffices to show smoothness of x ′ τ .
On U c , the construction implies that x ′ τ are smooth, so we now restrict to the event E .
We first compute the Jacobian of ψ :
[Equation]
and thus
[Equation]
which in turn implies:
[Equation]
i =
D ψ ( x i )
D ψ ( x i ) T (cid:33) .
Thus, under U , we have that
[Equation]
We may now use Proposition 43 to get that ( x τ 1 , . . .
, x τ p ) has a law that is ( βγσ/T ) p smooth with respect to µ ⊗ p d and apply Proposition 41 to get that, conditional on E , the law of ψ ( x τ 1 , . . .
, x τ p ) is σ ′ -smooth with respect to µ m , where
[Equation]
where we let V =
U and note
that P (( x τ 1 , . . .
, x τ p ) ∈
U ) ≥ 12 .
The result follows.
■
E.3.1 Proof of Propossition
We will proceed by using the co-area formula [Federer, 2014].
For a given x ∈ ( B d 1 ) n , let
[Equation]
[Equation]
and let H j denote the j -dimensional Hausdorff measure.
Then the co-area formula tells us that for any B ⊂ V , we have (cid:90)
[Equation]
We make use of the following lemma:
Lemma 44.
Suppose that ϕ is as in Proposition 41.
Then for all y ,
[Equation]
Before proving the lemma, we require a preliminary result from Yomdin [1984]; we reprove it here in order to keep track of the constant.
Lemma 45.
Fix k ∈ N and suppose that Y ⊂ B k 1 is a hypersurface and let B kr ⊂ B k 1 .
Suppose that any line passing through B dr intersects Y in at most ℓ points.
Then,
[Equation]
Proof.
Because B kr is convex, we may consider the map π : Y → ∂ B r of projection to the boundary.
Recentering so that B kr has center at the origin, we have π ( y ) =
r y
||
y || .
By the co-area formula introduced as (E.11), we have (cid:90)
[Equation]
Now, note that J ( π ( y )) ≥ r k and thus we have
[Equation]
where the last inequality comes from combining the fact that by assumption vol 0 ( ψ − 1 ( z )) ≤ ℓ and the expression for the surface area of S d − 1 .
■
Proof of Lemma 44.
We apply Yomdin [1984, Theorem 3 (iii)] iteratively on the coordinates of ψ .
In particular, we apply Lemma 45 in order to keep track of the explicit constant in Yomdin [1984,
Lemma 7] and apply Yomdin [1984, Lemma 4] to show that we may choose r = (20 · ℓ 2 ) − 1 in
Lemma 45.
Thus we have for any y ∈ B m 1 ,
[Equation]
using the fact that pd ≥ m .
The result follows.
■
Returning now to the proof of Proposition 41, we see for a given set B ⊂ V , that
[Equation]
where the first inequality follows from the definition of smoothness, the second inequality follows by (E.11) and the above claims [Equation] and the last inequality follows by the definition of µ m .
The result follows by using the fact that
[Equation]
and bounding (cid:0) m
[Equation]
E.3.2 Proof of Proposition
We first introduce a small-ball estimate for sums of PSD random variables, in the spirit of Simchowitz et al. [Equation]
Lemma 46.
Let X 1 , X 2 , . . .
, X p be i.i.d.
of positive semi-definite, R D × D -valued random variables, and suppose there exists B, η > 0 and Λ ∈ S D ++ for which, for all t ∈ [ p ] ,
[Equation]
Then, if p ≥ 8 η − 1 log
[Equation]
[Equation]
In particular
[Equation]
Proof.
The proof follows along the lines of Simchowitz et al.
[2018], sharpened slightly for the less general setting.
Let Σ = (cid:80) pi
=1 X i .
By a Chernoff bound (Lemma 17), for any v ∈ S D − 1 ,
P [ v ⊤ Σ v ≤ ηpv ⊤ Λ v/ 2] = P [
[Equation]
pi
[Equation]
1 v ⊤ X
i v ≤ ηpv ⊤ Λ v/ 2] ≤ P [ (cid:80) pi
{+=
 I { v ⊤ X+}
i v ≤ ηv ⊤ Λ v } ≤ ηp/ 2] ≤ exp( − ηp/ 8) , where we use that X i ⪰ 0 .
Hence, for any finite subset T ⊂ S D − 1 ,
[Equation]
To conclude, we show that there exists a finite set T of size at most exp( D 2 log( 12 Bη ) + 12 log det(Λ)) such that, if v ⊤ Σ v ≥ v ⊤ Λ v · ηT/ 2 for all T , then Σ ⪰ ηT 4 .
We take T to be an ε = (cid:112) η/ 4 B -net of the set S Λ := { v ∈ R d : v ⊤ Λ v = 1 } .
Then, if v ⊤ Σ v ≥ v ⊤ Λ v · ηp/ 2 = ηp/ 2 for all ˜ v ∈ S Λ , it holds
[Equation]
which means that Σ ⪰ ηp 4 Λ .
Define the ellipsoid E Λ := { v ∈ R d : v ⊤ Λ v ≤ 1 } ⊃ S Λ .
Note that since λ max (Λ) ≤ B , E Λ ⊃ { v : ∥ v ∥ 2 ≤ 1 /B } ⊃ 2 ε B D 1 , since ε = (cid:112) η/ 4 B ≤ 1 2 √ B .
[Equation]
vol (cid:0)
[Equation]
(cid:18)5 ε (cid:19) D det(Λ − 1 / 2 ) .
Hence, we can take
[Equation]
Lemma 47.
Let Ψ : R d → S D + be any function whose image are PSD matrices whose entries are polynomials of degree at most ℓ .
Let ρ be any distribution which is σ -smooth with respect to a log-concave measure µ .
Then, there exist a universal constant C such that, for any v ∈ R D \ 0 ,
[Equation]
Proof.
Consider the polynomial function f v ( x ) = v ⊤ Ψ( x ) v .
This is a polynomial of degree ℓ in x , and nonnegative.
By Carbery and Wright [2001, Theorem 8], with q = ℓ , we have
[Equation]
where C ′ is a universal constant.
Reparametrizing α ← α · E x ∼ µ [ f v ( x )] , we have
[Equation]
To conclude, take α = (2 C ′ ℓ/σ ) − ℓ , we get
[Equation]
Hence,
[Equation]
Taking C = 2 C ′ and substituting f v ( x ) = v ⊤ Ψ( x ) v concludes.
■
Proof of Proposition 42.
Let ρ t denote the conditional distribution of x t | x 1 , . . .
, x
t − 1 .
By assumption, ρ t is σ -smooth with respect to the log-concave measure µ , so
[Equation]
Hence, we can apply Lemma 46 with η ← 1 / 2 , B ← B , and Λ ← ( Cℓ ) − ℓ Λ .
Using that det(( Cℓ ) − ℓ Λ) =
( Cℓ ) − Dℓ det(Λ) concludes.
■
F Proofs from Section
In this section, we prove the extensions of our results to the multipiece setting.
F.1 Proof of Theorem
Algorithm Description.
Algorithm 6 gives our algorithm for K -class classification.
We maintain (cid:0) K 2 (cid:1) instances of the binary classification algorithm, Algorithm 1.
That is, each A bin maintains a w ( i,j ) t at each time t , and
[Equation]
To gain intuition, recall that we assume the ground-truth classifier to
[Equation]
where the argmax is taken lexicographically.
Hence, f ⋆ ( x ) admits the following equivalent representation:
[Equation]
Hence, f ⋆ ( x ) can be thought of running a lexicographic tournament, picking out the first index i which ‘wins’ over all lesser indices k .
This is what motivates the selection of ˆ y in Appendix F.1 of
Algorithm 6.
Proof of Theorem 12.
We reduce to the generalized, “censored” variation of our linear classification setting, depicted in Proposition 33.
For pairs i < j , define
[Equation]
i
[Equation]
Note that
[Equation]
For simplicty, lets assume that w ⋆, ( i,j ) ̸ = 0 for i < j .
We address the edgecase where this term may be zero at the end.
Further, let i t <
j t denote the indices select in Equation (F.4).
Then, since the algorithm always selects such a pair ( i t , j t ) whenever a mistake is made (and defining, say ( i t , j t )
= (0 , 0) to indicate no mistake),
[Equation]
The following claim reduces to binary-losses.
Claim 2.
For the indices i t < j t selected in Equation (F.4) , and any 1 ≤ i < j ≤ K ,
[Equation]
Moreover, when ( i t , j t ) =
( i, j ) , y ( i,j ) t =
sign(ˆ y t − y t ) , and thus can be determined by learner.
Proof.
Indeed, at a round where I { ( i t , j t ) =
( i, j ) } , we have ˆ y t ̸
= y t .
We have two cases •
When y t < ˆ y t , then Equation (F.4) selects i =
y t and j as some index for which ˆ y ( i,j ) t = − 1 , such an index must exist by the choice of ˆ y t in Appendix F.1 (otherwise, either ˆ y t <
y t , or else y t would be correctly selected as the true class).
On the other hand, y ( i,j ) t := sign( ⟨ w ( i,j ) ⋆ , x t ⟩ ) = 1 = sign(ˆ y t − y t ) by Equation (F.3).
Thus, y ( i,j ) t ̸ = ˆ y ( i,j ) t
• If ˆ y t < y t , then from Appendix F.1 it must be the case
that ˆ y
( i,j ) t = 1 for i =
ˆ y t and j = y t being the indices selected in Equation (F.4).
But by the reverse of Equation (F.3), y ( i,j ) t = − 1 = sign(ˆ y t − y t ) .
Hence, ˆ y ( i,j ) t ̸ =
y ( i,j ) t .
■
Hence, we may write
[Equation]
We now claim that the losses ℓ ( i,j ) t
[Equation]
i
[Equation]
) } precisely corresponding to the censored binary setting of Proposition 33.Indeed, consider a setting where x 1 , x 2 , . .
. are selected by the σ -smooth adversary, and the label is ˆ y
( i,j ) t defined above.
A ( i,j ) bin does not always see ˆ y
( i,j ) t , but whenever ℓ ( i,j ) t = 1 , Claim 2 shows that the learner does indeed observe the true value ˆ y ( i,j ) t .
Thus, by Proposition 33, it holds for any fixed i < j that with probability 1 − δ ,
T
[Equation]
Union bounding over all (cid:0) K 2 (cid:1) ≤ K 2 pairs i < j and summing, we conclude that with probability 1 − δ ,
Reg T = (cid:88)
[Equation]
Modification for non-unique ground truth classifiers.
Here, we can modify A ( i,j ) bin with the following rule: predict ˆ
y
( i,j ) t = 1 until there is an time t for which ( i t , j t )
=
( i, j ) , and then reinitalize A ( i,j ) bin to have w ( i,j ) t = e 1 , as in Algorithm 1.
Consider an i < j with w i⋆ = w j⋆ .
We claim ( i t , j t ) ̸
= ( i, j ) for any t .
Now, suppose there is a time t that ( i t , j t )
=
( i, j ) , let τ denote the first time t for which this is true.
Then, ˆ y ( i,j ) t = 1 .
But in addition y t ̸ = j for any t because we assume the arg max in Equation (F.1) is broken lexicograophically.
Thus, from Equation (F.4), it must be that y τ
= i , and that j is such that ˆ y
( i,j ) t = − 1 ; this gives a contradiction.
Now consider i < j with w i⋆ ̸ = w j⋆ .
Then our modification of ˆ y ( i,j ) t only increases (cid:80) Tt =1 ℓ ( i,j ) t by at most 1 .
This adds at most (cid:0) K 2 (cid:1) <
K 2 to the total regret (modifying the constant of 90 to 91 ).
■
F.2 Formal Guarantees for Piecewise Regression
We will prove a slightly more general version of Theorem 13 and then derive the result in Sectionas a corollary.
First, we will define what kinds of regression classes our result will apply to:
Definition 48.
Let G : X → R be a function class.
We say that G is ℓ -determined with respect to some measure µ on X if the following two conditions hold:
• The values on ℓ points in general position uniquely determine the function,
i
e., P ( there exist g ̸ =
g ′ ∈
G such that g ( x i ) =
g ′ ( x i ) for 1 ≤ i ≤ ℓ and x i ∼ µ ) = 0 (F.5) • Two functions intersect only on measure zero sets,
i
e., for all g, g ′ ∈ G ,
[Equation]
1: Initialize A do 3: do 5: predict
6: if y = y t then ( % self . errorUpdate (
Define (
[Equation]
Update
Note that linear classes in R d are trivially d -determined with respect to the Lebesgue measure, and thus with respect to any measure absolutely continuous with respect to the Lebesgue measure.
Polynomial classes are also ℓ -determined with respect to the Lebesgue measure for some ℓ depending on d and the degree of the polynomials.
We observe that our definition of an ℓ -determined function class is an offline analogue to the notion of eluder dimension from Russo and Van Roy [2013].
Now, for a given function class G : X → R , we denote by
[Equation]
where F is the set of K -class linear classifiers from Theorem 12.
We will continue to suppose that the x t are drawn from distributions that are σ -smooth with respect to µ and that the labels y t are realizable with respect to G F .
Assumption 1 (Oblivious, realizable smoothed sequential setting) .
We suppose smoothed online learning setting and the adversary is realizable with respect to G F and oblivious in the sense that before the learning process begins, the adversary chooses g ⋆ =
( g ⋆ 1 , . . .
, g ⋆K ) ∈
G K and f ⋆ ∈ F and lets y t
= ( g ⋆ )
f ⋆ ( x t ) for all t .
We assume further that g ⋆ has unique entries: g ⋆i ̸
= g ⋆j for i ≤ j .
Lastly, we assume we have access to the following ERM oracle.
Definition 49 (ERM Oracle) .
Given U
= { ( x 1 , y 1 ) , . .
, ( x n , y m ) } , where ( x i , y i )
∈ B d 1 × Y ,
ERM ( U , G , K ) returns a n ≤ K , and g 1 , . .
. , g n and partition C 1 , . .
, C n of U
such that
, for all ( x, y ) ∈
C i , g i ( x ) = y .
By post-processing, we may also assume that g i are distinct
Proposition 50 (General ℓ -Determined Regression) .
Suppose that we are in the semi-oblivious, smoothed online learning setting, where the adversary begins by choosing g ⋆f ⋆ G F from (F.7) , and, at each time t , draws x t from a distribution that is σ -smooth with respect to µ and sets y t =
g ⋆ f ⋆ ( x t ) .
Suppose further that G is ℓ -determined, in the sence of Definition 48.
Then, Algorithm 7 satisfies for all T , with probability at least 1 − δ ,
[Equation]
(F.8) Moreover, the per-time step computational complexity of Algorithm 7 is polynomial in d and the complexity of the ERM oracle Definition 49, applied to a data set U of size no more that | U | ≤
K ( ℓ + 1) .
F.3 Algorithm for Piecewise Regression
Algorithm 7 proceeeds at follows.
We let N t denote the number of clusters about which we are certain, U t denote the set of points which cannot be assigned to a cluster.
We maintain a supervised
1: Init: K -class supervised linear classifier A (instance of Algorithm 8)
ERM-oracle ERM (see ) 2: for each time t = 1 , 2 , . . .
do 3: recieve x t 4: predict ˆ y t
= ˆ g k ( x t ) for k
= ˆ k t , where ˆ k t :=
A .
classify ( x t , N t ) %
ˆ y t
= 0 if N t =5: observe y t .
6: if ∃ k ⋆t ∈ [ N t ] with ˆ g k ( x t ) =
y t then % update classification 7: if ˆ k t ̸ =
k ⋆t then , A .
errorUpdate ( x t , N t ) 8: maintain N t +1 ← N t , U t
+1 ← U t 9: else % update clustering 10: ( C 1: n , g 1: n ) ← ERM
(˜ U t , G , K ) , ˜ U t =
U t ∪ { ( x t , y t ) } % Initialize ˜ N = N t 11: for each i : | C i | ≥ ℓ + 1 do 12: ˜ N = ˜ N + 1 , ˆ g ˜ N ← g i , 13:
N t +1 ← ˜ N , U t +1 ← ˜
U t \ (cid:83) i : | C i |≥ ℓ +1 { ( x, y ) ∈ ˜
U t : g i ( x ) = y }
1: Initialize Binary classifiers A ( i,j ) bin , i < j 2: for t = 1 , 2 , . . .
do % guarantee y t ≤ M t 3:
Recieve ( x t , M t ) and predict ( % self .
classify ( x t , M t ) )
ˆ y t =
min { i ∈
[ M t ] : A ( i,j ) bin .
classify ( x t )
= 1 , i < j ≤ M t } , 4:
Observe y t 5: if ˆ y t ̸
= y t then ( % self .
errorUpdate ( x t , M t ) ) 6:
Define ( i, j ) =
(cid:40)
i =
y t , j ∈ { k > i : ⟨ w ( i,k ), x t ⟩ < 0 } if y t <
ˆ y j
i
=
ˆ y t , j = y t
if ˆ y j < y t (F.9) 7:
Update A ( i,j ) bin . errorUpdate ( x t )
K -class linear classifier, A , described in Algorithm 8.
It is similar in spirit to Algorithm 6, except it takes in “side information” M t on which it only predicts from the first M t classes.
Lastly, we maintain a growing sequence of regressors ˆ g 1 , ˆ g 2 , · · · ∈
G such that ˆ g i does not change once assigned, and ˆ g i is defined for all i ≤ N t .
At each time t , we call ˆ k t =
A .
classify ( x t , N t ) to guess the cluster of x t , only among cluster i ≤ N t about which we are certain.
Then, we predict ˆ y t
= ˆ g ˆ k t ( x t ) .
The idea is that, for k
= ˆ k t ≤ N t , we are sure that ˆ g k is the true predictor if x t is in cluster k .We
then observe y t .
If y t was correctly predicted by one of that ˆ g i for which i ≤ N t , but not the ˆ k t we guessed, then we update our classifier A .
Otherwise, we call the ERM oracle to determine if we can find new cluster(s) to add, appending to our sequence of predictors ˆ g ’s, and growing our number of certain clusters N t .
Note that we never maintain an explicit clustering of our points, but only cluster retroactively based on whether ˆ g i ( x t ) =
y t for some i , as a means to recover the classification label.
F.4 Proof of PropositionF.4.1 Guarantee for ERM procedure
Lemma 51.
Let I ⊂ [ T ] be any subset of time.
Then with probability one, it holds that for any partition C 1 , . . .
, C n of ( x s , y s ) s ∈ [ I ] and any g 1 , . . .
, g n distinct functions such that, for all ( x, y ) ∈ C n , ˜ g
i ( x ) =
y , then for any index i for which | C i | ≥ ℓ + 1 ,
• f ⋆ ( x ) =
f ⋆ ( x ′ ) for all ( x, y ) , ( x ′ , y ′ )
∈ C i • ˜ g
i =
g ⋆f ⋆ ( x ) , representative x ∈ C
i
Proof.
Let I 1 , . .
, I m denote the times in each cluster C 1 , . . .
, C n .
Without loss of generality, suppose I 1 is a cluster for which |
I 1 | ≥ ℓ + 1 (we may handle all simultaneously via a finite union bound.)
Item 1.
Suppose in fact that there exists s, s ′ ∈
I 1 with f ⋆ ( x s ) ̸ =
f ⋆ ( x s ′ ) .
We first argue then that g 1 ̸
= g ⋆k for all k ∈
[ K ] .
Indeed, by smoothness and the second condition of Definition 48, it holds that with probability 1 , g ⋆k ( x ˜ s ) ̸ =
g ⋆k ′
( x ˜ s ) for all 1 ≤ ˜ s ≤ T and k ̸ = k ′ .
Set i
= f ⋆ ( x s ) and i 2 = f ⋆ ( x s ′ ) .
Thus, if g i = g
⋆k , the fact g i ( x s ) =
g ⋆f ⋆ ( x s ) and g i ( x s ′ )
=
g ⋆f ⋆ ( x s ′ ) would require both g ⋆k ( x s ) = g ⋆f ⋆ ( x s )
=
g ⋆i 1 ( x s ) and g ⋆k ( x s ′ ) =
g ⋆f ⋆ ( x s ′ )
=
g ⋆i 2 ( x s ′ ) .
Thus, on the aforementioned probability one event, we would have both k
= i 1 and k
= i 2 , which contradicts the supposition i 1 ̸ =
i 2 .
Next, let S ⊂ [ T ] denote a set of indices.
Denote s max = max { s ∈ S } , and define the events
[Equation]
By the above observation that g i ̸
= g ⋆k for any k , we see that if there exists s, s ′ ∈
I 1 with f ⋆ ( s ) ̸ =
f ⋆ ( s ′ ) with | I 1 | ≥ ℓ + 1 , then one of the events A S ( g ⋆k ) must occur for some | S | ≥ ℓ +and k ∈
[ K ] .
Since there are only finitely many such events, it suffices to show that for any fixed S and k , P
[ A S ( g ⋆k )]
= 0 .
Hence, fix S and k .
For a given S with max element s max , let F − 1 denote history generated by ( x 1 , y 1 ) , . . .
, ( x s max − 1 , y s max − 1 ) .
Define the A − 1 :
= {∃ g ∈ G \ { g ⋆k } : s ∈ S, g ( x s ) =
y s , s ∈
S \ { s max }} .
Then, A − 1 is F − 1 measurable
and A − 1 contains A S ( g ′ ) .
Hence,
[Equation]
By the first condition of Definition 48, A − 1 coincides with the event A ′− 1 :=
{∃ a unique g ∈ G \ { g ⋆k } : s ∈ S, g ( x s ) =
y s , s ∈ S \ { s max }} almost surely.
Hence,
[Equation]
Lastly, when A ′− 1 holds, let ˆ g ̸ =
g ⋆k denote the unique g ̸ =
g ⋆k consistent with examples s ∈
S \ { s max } .
Since ˆ g is determined by F S − 1 , we have
[Equation]
where we use that ˆ g is fixed , that ˆ g ̸ = g ′ , and the second condition of Definition 48.
The bound follows.
Item 2.
For any fixed set of indices ˜ I with | ˜
I | ≥ ℓ + 1 ≥ ℓ , the first condition of ℓ -determination (Definition 48) ensures then that P [ ∃ g 1 ̸
= g ⋆j : g 1 ( x s ) =
g ⋆j ( x s ) , ∀ x ∈ ˜ I ] = 0 .
The bound follows by union boundig over all ˜ I
⊂ [ T ] and j ∈ [ K ] .
■
F.4.2 Distinctness of clustering
Claim 3 ( U t is Uncertain Set) .
Fix a time t .
Then, for any ( x, y ) ∈
U t and any i ≤ N t ,
ˆ g
i ( x ) ̸ = y .
Proof.
This is true vacuously at time t = 1 , when U t = ∅ .
Suppose it holds at time t , we prove it for time t + 1 .
If x t is such that there exists an n ≤ N t with ˆ g i ( x t ) =
y t , then U t +1 does not change from U t .
Otherwise, if ˆ g n ( x t ) ̸
= y t for all n ≤ N t ,
U
[Equation]
←
[Equation]
where g i and C i are the clustering from the ERM oracle.
By the inductive hypothesis and fact that ˆ g n ( x t ) ̸
= y t for all n ≤ N t , it follows that ˆ g n ( x ) ̸
= y for all
( x, y ) ∈
U t ∪ { ( x t , y t ) } = ˜
U t ⊇ U t +1 .
Now, if there is some n : N t < n ≤ N t for which ˆ g n ( x ) =
y , then that ˆ g n was added during the
ERM step at round t : i.e. ˆ g n = g i for some i such that | C i | ≥ ℓ + 1 .
But then ( x, y ) is removed form U t +1 by Equation (F.10).
■
Claim 4.
Fix a time t .
Then, for any i, j ≤ N t , ˆ g i ̸
= ˆ g j .
Proof.
This is trivially true at time t = 1 .
Suppose this is true at time t , we establish the claim for time t + 1 .
If x t is such that there exists an i ≤ N t with ˆ g i ( x t ) =
y t , then N t +1 = N t
and so the set of ˆ g
i ’s under consideration remains unchanged.
On the other hand, suppose there is no i ≤ N t with ˆ g i ( x t ) =
y t .
Then, all possible new ˆ g j ’s for
N t <
j ≤ N t +1 are correct on some subset of points of U t ∪ ( x t , y t ) .
But by the previous claim (Claim 3) and the assumption that, for i ≤ N t with ˆ g i ( x t ) =
y t , no element of U t ∪ ( x t , y t ) is correctly predicted by any ˆ
g
i for i ≤ N t .
Thus, none of the new ˆ g j ’s can equal an ˆ
g i for i ≤ N t .
Moreover, by the definition the ERM oracle, Definition 49, all newly added ˆ g j ’s are distinct.
■
F.4.3 Key summary of Algorithm
We now summarize the results with the following lemma.
Lemma 52.
With probability 1 , there exists a permutation π such that
• For each time t and i ∈ [ N t ] , ˆ g i =
g ⋆π ( i ) • For each time t and i ∈
[ N t ] , ˆ g i ( x ) =
y if and only if f ⋆ ( x ) =
π ( i ) •
If ( x, y ) ∈
U t , then π − 1 ( f ⋆ ( x )) > N t .
Whenever ˆ y t ̸ = y t , either π − 1 ( f ⋆ ( x )) > N t , or ˆ k t : =
A .
classify ( x t , N t ) has π (ˆ k t ) ̸
= f ⋆ ( x t ) .
Proof.
For n = N T , let ˆ g 1 , . . .
, ˆ g n ∈ G denote the functions constructed by our algorithm.
Since each new ˆ g i is added from a cluster with at least ℓ +1 points, applying Lemma 51 (with a union bound over index sets I ensures that ˆ g
i = g ⋆j ) for some j ∈ [ K ] .
This gives us a mapping π : [ n ] → [ K ] .
π must be injective, since g ⋆j are distinct by assumption, and ˆ g i are unique by Claim 4 (in particular, n ≤ K ).
Thus, π can be extednded to a permutation from [ K ] → [ K ] .
By construction, the first item is satisfied.
The second item is a consequence of uniquenessof that the previous point, uniqueness of ˆ
g
i ’s, and the second point of Definition 48 , since we only need to union bound over finitely many times t ∈
[ T ] and pairs g ⋆i , g ⋆j ’s.
The third item follows similarly, by invoking Claim 3.
For the last point, suppose π − 1 ( f ⋆ ( x t )) ≤ N t .
Then, by the previous point, ( x t , y t ) / ∈ U t .
Thus, the algorithm classifies ˆ y t
= ˆ g ˆ k t ( x t ) where ˆ k t ∈ [ N t ] .
But by the first point of the lemma, ˆ g ˆ k t = g ⋆π (ˆ k t ) .
So if π (ˆ k t ) =
f ⋆ ( x t ) , then we woudl have ˆ g ˆ k t ( x t ) =
g ⋆f ⋆ ( x t ) ( x t ) =
y t , a contradiction.
■
F.4.4 Proof of Proposition
Let π denote the permutation ensured by Lemma 52.
We may assume without loss of generality that π is the identity permutation (by permuting g ⋆ ).
Let k t = f ⋆ ( x t ) .
Recalling also that ˆ k t ≤ N t , the fourth point of Lemma 52 ensures.
[Equation]
First, we bound the contribution of I { k t > N t } :
Claim 5.
(cid:80)
Tt
=
I { k t = k, k > N t } ≤ K ( ℓ + 1) .
Thus, (cid:80) Tt
=
I { k t > N t } ≤ K 2 ( ℓ + 1) .
Proof.
Suppose k > N t , and let S t,k := { s ≤ t : f ⋆ ( x s )
= k } , and define τ k = max { t ∈ [ T ] : k t > N t , k t = k } .
Then
[Equation]
We claim that | S τ k ,k | ≤ K ( ℓ + 1) .
Indeed, suppose | S τ k ,k | > K ( ℓ + 1) .
Then, for some t < τ k , | S t,k | =
K ( ℓ + 1) and k t = k and k > N t .
By Lemma 52, g ⋆k t ( x t ) ̸
= ˆ g i ( x t ) for any i ≤ N t .
Hence, our algorithm executes Appendix F.3.
By the pidgeon-hole principle, there must be at least one cluster
C i : | C i
| ≥ ℓ + 1 which contains at least one s ∈ S t,k .
Hence, the update rule ensures that i ≤ N t + for which
ˆ
g
i
( x s )
=
y s .
But again, by Lemma 52 (and taking the permutation to be the identity), we have f ⋆ ( x s ) = i .
In other words, i = k s = k , i.e. k s ≤ N t
+1 ≤ N τ k .
This constradictions the definition of τ k .
■
Summarizing our argument thus far, the following holds with probability one
[Equation]
Finally, by mirroring the proof of Theorem 12, we upper bound
[Equation]
The key difference between the above bound and that of Theorem 12 is that we only see when get feedback k t ≤ N t , but at the same time, we only suffer a loss when k t ≤ N t .
Hence, the bound follows from a near-identical argument, calling the general censored version of our binary classification Proposition 33, modified to add the event { k t ≤ N t } to the censoring.
Combining
Equations (F.11) and (F.12) concludes.
■
G Non-realizable mistake bounds for the Perceptron.
For simplicity, we consider regret with respect to a fixed b ⋆ ∈ R , w ⋆ ∈ B d 1 , and define y ⋆t =
y ⋆ ( x t ) , y ⋆ ( x ) : = sign( b ⋆ + ⟨ x t , w ⋆ ⟩ ) .
Again, we normalize x t that max t ∥ x t ∥ ≤ 1 .
We further assume that
[Equation]
We show in Lemma 59 at the end of this section that this is without loss of generality.
We define ˆ w ⋆ = [Equation] ∥
[Equation]
⋆ ∥ .
Unlike with our cutting-plane methods, we allow the adversary to deviate from a realizable clasiffier.
Specifically, for each time t , the adversary selects x t ∼ p t , and may instead choose to play some y t ̸ = y ⋆t .
We define,
[Equation]
and obtain non-vacuous mistake bounds provided N err is sublinear in T .
Informally, our total mistake bound for the Perceptron is polynomial in the smoothness along the direction of the optimal classifier ˆ w ⋆ .
This is formalized in the following definition:
Definition 53 (Directional σ dir -smoothness) .
We say that the adversary
• is ( σ dir , ˆ w ⋆ ) directionally-smooth if ⟨ x t , ˆ w ⋆ ⟩ has density at most 1 /σ dir with respect to the Lebesgue measure on the real line.
• is, more generally, ( σ dir , α, ˆ w ⋆ ) directional-Tsybakov-smooth if sup a ∈ R P x t ∼ p t [ ⟨ x t , ˆ w ⋆ ⟩ ∈ [ a, a + η ]] ≤ η 1 − α /σ dir .
Note that a ( σ dir , α, ˆ w ⋆ ) -Tsybakov adversary is is ( σ dir , ˆ w ⋆ ) -smooth.
Note that Definition 53 is a slightly weaker condition than the one consider in Theorem 7 in the body, as it only requires directional smoothness along ˆ w ⋆ (not uniformly).
As noted in the body, directional smoothness can differ substantially from general smoothness.
We provide two examples.
Example 1 (Additive d -Ball Noise) .
Suppore that at each time t , the adversary selects x t =
ˆ x t + e t , where ∥ ˆ x t ∥ ≤ 1 / 2 , and e t ∼ r B d 1 for r ≤ 1 / 2 (and, for simplicity, d > 1 ).
Then, the adversary is σ -smooth for σ = vol d ( r B d 1 ) / vol d ( B d 1 ) =
r d .
However, if u ∼ µ d is drawn uniformly from the sphere, then the density p 1 ( · ) of its first coordinate u 1 with respect to the Lebesgue measure is
[Equation]
Hence, by rotational symmetry, we see that for any w ⋆ , the adversary is ( σ dir , ˆ w ⋆ ) directionallysmooth for σ dir = 2 √ πr ( d − 1) .
Notice that the directional smoothness is now only polynomial in d , rather than exponential in it.
Example 2 (Additive Noise in a Random-Direction) .
Again consider the additive noise setting where at each time t , the adversary selects x t = ˆ x t + e t , where ∥ ˆ x t ∥ ≤ 1 / 2 .
However, suppose e t is selected as follows: before the game, the adversary selects a direction ˆ e ∼ µ d , and plays e t = a t ˆ e , where a t is drawn uniformly on the interval [ − r/ 2 , r/ 2] .
Note that this adversary need not be σ -smooth with respect to µ d for any σ > 0 , because after the adversary commits to ˆ e , her smoothing is restricted to a line segment.
Still, with constant probability, ⟨ ˆ e, ˆ w ⋆ ⟩ ≥ c/d for some constant c > 0 .
Hence, with constant probability, the adversary is ( σ dir , ˆ w ⋆ ) -directionally smooth for σ dir = cr/d .
We now state our guarantee for the classical Perceptron algorithm Rosenblatt [1958]
Theorem 54.
Suppose that the adversary is ( σ dir , α, ˆ w ⋆ ) -Tsybakov, and define ρ := 2 3 − α ∈
Then, with probability 1 − δ , the Perceptron algorithm (Algorithm 9) satisfies
[Equation]
1: Initialize
= e
B d2: for
do 3:
Recieve
←
Remark 4.
Recall Example 1, which shows that directional smoothness σ dir may scale as ∼ r/d when the (standard) smoothness scales as σ = r d .
Applying Theorem 54 with α = 0 and thus, ρ = 2 / 3 , our ( T/σ dir ) 2 / 3 ∼ ( Td/r ) 2 / 3 -mistake bound interpolates between the log( T/σ ) ∼ d log(1 /r )
+ log( T ) bounds attained in this paper, and the poly(1 /σ ) ∼ (1 /r )
Ω( d ) -regret enjoyed by previous computationally efficient algorithms.
In addition, we achieve a robustness to sublinearlyin- T mistakes, which prior approaches do not.
In fact, a more general result holds, in terms of a direction-wise anti-concentration of the adversaries distributions.
Theorem 55 (Guarantee under Tsyabkov Smoothness) .
Define the anti-concentration function
[Equation]
For any fixed γ ∈ (0 , R ) , with least 1 − δ , the number of mistakes made by the Perceptron (Algorithm 9) is at most T (cid:88)
[Equation]
G.1 Proofs for the Perceptron
We begin by stating the standard guarantee for the Perceptron algorithm due to Freund and Schapire [1999].
To emphasizes its generality, we use ¯ x i to denote its inputs, which we allow to have non-normalized radius R .
Theorem 56.
Let (¯ x i , y i )
Ti =1 ∈ R n × R be a sequence of labeled exampled with ∥ ¯ x i ∥ ≤ R .
Fix ¯ w ∈ S n − 1 , γ > 0 , and define the margin errors
[Equation]
Then, the number of mistakes make by the online Perceptron is at most
[Equation]
The following corollary explicitly bounds the term D 2 ,
Corollary 57.
Fix a ¯ w ∈ S n − 1 , γ ∈ (0 , R ) .
Let
[Equation]
Then, the number of mistakes make by the online Perceptron is at most
[Equation]
Proof of Corollary 57.
Let S 1 :
= { i : sign( y i · ⟨ ¯ x i , ¯ w ⟩
) ̸ = 1 } and S 2 :
= { i : sign( y i · ⟨ ¯ x i , ¯ w ⟩ ) = 1 , y i · ⟨ ¯ x i , ¯ w ⟩ ≤ γ } .
Note that N 1 := | S 1 |
and N 2 :=
| S 2 | .
Moreover, S 1 ∩ S 2 = ∅ , and if i / ∈ ( S 1 ∪ S 2 ) , d i := max { 0 , γ − y i · ⟨ ¯ x i , ¯ w ⟩} = 0 .
Hence,
[Equation]
For i ∈ S 1 , d 2 i ≤ ( γ + |⟨ ¯ x i , ¯ w ⟩| ) 2 ≤ ( γ + R ) 2 ≤ 4 R 2 , where we used γ ≤ R .
For j ∈ S 2 , d j ∈ [0 , γ ] , so d 2 j ≤ γ 2 .
Thus, D 2 ≤ 4 R 2 N 1 + N 2 .
Thus, ( R
[Equation]
We now return to our specific setting, re-adopting x i (not ¯ x i ) for features.
We bound the probability that a given point x i does not lie within a margin γ .
Lemma 58.
Consider the p µ function from eq.
( % self .
classify ( x t ) ) .
Then, for any interval I 0 ⊂ R ,
[Equation]
In particular, P [ y ⋆t · ( b ⋆ + ⟨ x t , w ⋆ ⟩ ) ≤ γ } | F t − 1 ] ≤ 2 p µ (2 γ, ˆ w ⋆ ) .
Proof of Lemma 58.
Note that the ground truth label y i may depend on x i .
We circumvent this with a union bound.
Let I 0 be any inverval.
[Equation]
where we recall our assumption ∥ w ⋆ ∥ ≥ 1 / 2 .
■
We may now prove Proof of Theorem 55.
Proof of Theorem 55.
We apply Corollary 57 with ¯ w = ( w ⋆ , b ⋆ ) and ¯ x i =
( x i , 1) .
Note then that ∥ ¯ x i ∥ 2 = 1 + ∥ x i ∥ 2 ≤ 2 , so we may take R = √ 2 .
Define
[Equation]
it suffices to bound N 1 and N 2 .
Sice y ⋆t · ⟨ ¯ x t , ¯ w ⟩ ) ≥ 0 , we see that each
N [Equation]
T (cid:88)
[Equation]
[Equation]
∈
[Equation]
γ ] } . Set t γ := 2 p µ (2 γ, ˆ w ⋆ ) + 8 log(1 /δ )
/m .
By Lemma 58,
[Equation]
Hence, by Lemma 17,
[Equation]
Thus, from Corollary 57, applyied to the vectors ( x t , 1) , the number of mistakes is at most
[Equation]
+ 4) R 2 γ 2 + 8 T p µ (2 γ, ˆ w ⋆ ) + 32 log(1 /δ ) .
■
Proof of Theorem 54.
Fix any N ∈ N .
Under the Tsybakov smoothness of the adversary,
[Equation]
Balance both terms by setting γ 3 − α N = ( N ) / ( σ − 1 dir T ) .
Then, for ρ = 2 3 − α , this choice of γ ensures
[Equation]
Since γ 3 − α N balanced the terms in Equation (G.1), we have
[Equation]
For k ∈ N , let E k := { 2 k − 1 ≤ N err ≤ 2 k } .
Then, P [ (cid:83) ⌈ log T ⌉ k
= ] = 1 .
Moreover, by applying
Theorem 55 with γ 2 k for each k , we have with probability 1 − δ , if E k holds, then applying
Equation (G.2) with N = 2 k ,
[Equation]
where in the last line, we use N err ≥ 2 k / 2 on E k .
Taking a union bound over k ∈ [ ⌈ log T ⌉ ] , with probability 1 − δ , # mistakes ≲ ( T/σ dir ) ρ ( N err ) 1 − ρ + log( ⌈ log T ⌉ /δ ) .
■
G.2 Lower bound on 1 / ∥ w ⋆ ∥
Lemma 59.
There exists ( ˜ w, ˜ b ) for which P [ y ⋆ ( x t ) =
sign(˜ b + ⟨ x t , ˜ w ⟩ ) , ∀ t ≥ 1] = 1 , and which satisfy | ˜ b | + | ˜ w | 2 = 1 , and ∥ ˜ w ∥ ≥ 1 / 2 .
Proof.
We consider two cases.
• Case 1: y ⋆ ( x ) is not constant on B d 1 .
Let (˜ b, ˜ w ) be equal to α ( b ⋆ , w ⋆ ) , where α is chosen so that ˜ b 2 + ∥ ˜ w ∥ 2 = 1 .
By positive homegenity of sign , y ⋆ ( x ) = sign(˜ b + ⟨ x t , ˜ w ⟩ ) .
Since y ⋆ ( x ) is not constant on B d 1 , we must have ∥ ˜ w ∥ ≥ | ˜ b | .
This means ∥ ˜
w ∥ 2 ≥ ˜ b 2 = 1 − ∥ ˜ w ∥ 2 .
Hence, ∥ ˜ w ∥ 2 ≥ 1 / 2 .
• Case 2: Since y ⋆ ( x ) ≡ y ⋆ is constant on B 1 d .
For some ε small, set ˜ b
= (cid:112) 1 / 2 + εy ⋆ , and set ˜ w = e 1 (cid:113) (1 − ˜ b 2 ) =
(cid:112) (1 / 2 − ε )
e 1 , where e 1 is the first cannonical basis vector.
By construction, ˜ b
2 + ∥ ˜ w ∥ 2 = 1 , and y ⋆ (˜ b + ⟨ x, ˜ w ⟩ ) ≥ (cid:112) 1 / 2 + ε − (cid:112) 1 / 2 + ε > 0 .
To conclude, we take ε = 1 / 4 (though any ε arbitrarily close to zero would work as well).
■